)i[ jddlZddlZddlZddlmZmZmZmZddlm Z m Z ddl m Z ddl mZddlmZddlmZddlmZdd lmZmZdd lmZdd lmZddlZdd lmZdd lm Z de!dee"fdZ#deedee"deefdZ$dde!dee"de!de!fdZ%GddZ&GddZ'GddZ(da)dZ*y)N)ListDictAnyOptional)ChatGoogleGenerativeAIGoogleGenerativeAIEmbeddings)MongoDBAtlasVectorSearch) MongoClient)Document)RecursiveCharacterTextSplitter)ChatPromptTemplate)RunnablePassthroughRunnableLambda)StrOutputParser)AsyncIOMotorClient)settings)extract_content prompt_textreturnchddl}|jd|}|rt|jSdS)zNExtract number from admin prompt like 'Please consider company providing 9...'rNz\d+)researchintgroup)rrmatchs K/var/www/html/Whatsapp_Customer_Support_Chatbot/src/services/rag_service.pyextract_admin_overriders- IIfk *E!&3u{{} 0D0docs admin_numberc|s|dk7r|Sgd}g}|D]_}|j}ddl}|D]"}|jd|dt||}$|j t ||j a|S)aFilter/modify document content based on admin override number. OPTION 1: Replace ALL policy numbers (not just specific keywords). When admin specifies a number like "0", replace all occurrences of common policy numbers (5, 6, 8, 10, 12, 15, 18, 20, 25, 30, etc.) r)  Nz\b page_contentmetadata)r0rsubstrappendr r1)rr policy_numbers filtered_docsdoccontentrold_nums rfilter_docs_by_admin_ruler:s LA- ENM "" % GffgYb!L!G  X \\   ( r system_prompt admin_textc 8|s|Sd|d|d|d|d }|dz|zS)z?Inject strict override instruction at the top of system prompt.zg ### CRITICAL ADMIN OVERRIDE (HIGHEST PRIORITY - ENFORCE STRICTLY): - Admin has specified: Use EXACTLY z items/holidays/values - If any document says a different number (like 12, 15, 20, etc.), that is OUTDATED - IGNORE the outdated number and ALWAYS use z- - Your response MUST consistently reference z[, not any other number - If asked about count/quantity in this domain, ALWAYS respond with  )r;r r<override_instructions rinject_strict_overriderA@sT  $$0>2--9N;--9N;EEQNS  $ & 66rceZdZdZdej fdZdefdZdefdZ dde jd e e fd Zd e fd Zd e fd Zy)DocumentIndexManagerz-Handles MongoDB Atlas Vector Search indexing.loggercR||_tdtj|_y)Nzmodels/gemini-embedding-001)modelgoogle_api_key)rDrrGEMINI_API_KEY embeddings)selfrDs r__init__zDocumentIndexManager.__init__Us! 6/#22 rrcBttj}|tjtj}|j idkDr|j jdn|j jdt||jdS)Nrz(MongoDB Atlas index found. Connecting...zFVector Database is empty. Use Admin API to upload and index documents. vector_index) collection embedding index_name) r r MONGODB_URIDB_NAMEVECTOR_COLLECTIONcount_documentsrDinfowarningr rI)rJclientrNs rload_or_build_indexz(DocumentIndexManager.load_or_build_index\sX112H,,-h.H.HI   % %b )A - KK  G H KK   h i'!oo%  rc <tjtj}|j st d|t |jd}|jjdt|dg}t|dD]\}}|jjd|dt|d|j tt|}|jsct!d d }|j#|}|D]*} |j%t'| d |ji ,|st'dd di g}|jjdt|dt-j.||j0t3tj4tj6tj8d} | S#t($r6} |jj+d|jd| Yd} ~ d} ~ wwxYw)NzData dir not found: z*.pdfz Indexing z documents to MongoDB Atlas.[/z] Processing: , chunk_size chunk_overlapsourcer/zError processing : zSystem active.systemz Uploading z chunks to Atlas...rM documentsrOrNrP)pathlibPathrDATA_DIRexistsFileNotFoundErrorlistrglobrDrUlen enumeratenamerr3stripr split_textr4r Exceptionerrorr from_documentsrIr rQrRrS) rJ data_path pdf_filesdocument_chunksindexpdf_file raw_contentsplitterchunkschunke vector_stores r rebuild_indexz"DocumentIndexManager.rebuild_indexlsLL!2!23 !#&:9+$FG G12  9S^$44PQR(A6 LOE8 KK  qqY(8x}}oV W L-c(m< "((*H9TY\]!,,[9#E#**8%*"*HMM!:, L$'5EQY[cPdefO :c/&:%;;NOP/>>%oo"8#7#789I9IJ8KeKef%   L !!$5hmm_Bqc"JKK Ls$G5A G H%+HHN file_path folder_namec tt|}|jsytdd}|j |}d|j i}|r||d<|Dcgc]}t ||j}}|rtj||jttjtjtjd |j j#d |j yycc}w#t$$r2} |j j'd |j d | | d} ~ wwxYw) z0Indexes a single PDF file into the vector store.Nr]r^r_rbrr/rMrezSuccessfully indexed zError indexing rc)rr3rqr rrrpr copyr rurIr rrQrRrSrDrUrsrt) rJrrr{r|r}r1r~rrs rindex_single_filez&DocumentIndexManager.index_single_files2 )#i.9K$$&5UXYH((5F )..1H*5'$ehmmoFD (77""oo*8+?+?@AQAQRS[SmSmn-     #88H!IJ   KK   /?r!E FG s/$D7D"D BD D E -EE filenamecp ttj}|tjtj}|j dd|id|igi}|j jd|jd|y#t$r(}|j jd|d||d}~wwxYw) z8Deletes all vectors associated with a specific filename.$orzmetadata.sourcerbDeleted z vectors for zError deleting vectors for rcN r rrQrRrS delete_manyrDrU deleted_countrsrt)rJrrWrNresultrs rdelete_file_vectorsz(DocumentIndexManager.delete_file_vectorss  !5!56F 0 01(2L2LMJ++&1x(-F KK  x(<(<'=]8*U V  KK   ;H:RsK LG BB B5 #B00B5cp ttj}|tjtj}|j dd|id|igi}|j jd|jd|y#t$r(}|j jd|d||d}~wwxYw) z;Deletes all vectors associated with a specific folder name.rzmetadata.folder_namerrz vectors for folder: z"Error deleting vectors for folder rcNr)rJrrWrNrrs rdelete_folder_vectorsz*DocumentIndexManager.delete_folder_vectorss  !5!56F 0 01(2L2LMJ+++[9"K0-F KK  x(<(<'==RS^R_` a  KK   B;-rRSQTU VG rN)__name__ __module__ __qualname____doc__loggingLoggerrKr rXrrgrhrr3rrrr?rrrCrCRsc7 w~~  %= %7%N7<<hsm<C rrCc^eZdZdedej fdZdZdefdZ d deded efd Z d Z y )RERAInferenceEnginerrDc||_||_ttjtj tj tj|_ttj|_ |jtj|_ d|_d|_i|_d|_d|_d|_y)N)rF temperaturemax_output_tokensrGr)rrDrr MODEL_NAME TEMPERATURE MAX_TOKENSrHllmrrQrWrRdb _cached_chain_last_prompt_text_response_cache _prompt_cache_prompt_cache_time_admin_override_number)rJrrDs rrKzRERAInferenceEngine.__init__s( )%% ,,&11#22   ))=)=> ++h../!!%!!"#&*#rcKddl}|jr,|j|jz dkr |jS |jdj ddij dd}|j d d{}dd lm}dd lm}|rW|Dcgc]&}|jd s|jd d (}}|r|jjdt|ddj|}t|}d} | dj|D cgc]} d|  c} z} |jd } t!fddDrd} |j#ddj#ddj#dd} | d| d| }|r4|jjd|d||_t'|||}nd|_||_n8||_d|_n)|jjd||_d|_|j|_|jS7cc}wcc} w#t($r7}|jj+d |dd lm}d|_|cYd}~Sd}~wwxYww)!zAFetch ALL active prompts with a 5-minute cache to reduce DB load.rNr^prompts is_activeT updated_atrZd)length)DEFAULT_SYSTEM_PROMPTr8zUsing z active prompts from DB.r>zE### ABSOLUTE PRIORITY: CUSTOM ADMIN RULES (OVERRIDE ALL OTHER RULES) z- c3&K|]}|v ywrr?).0wordprompt_text_lowers r z?RERAInferenceEngine._get_system_instructions..sg4#44gs)ignorezdo notzdon'tonlya> STRICT COMPLIANCE RULE: If any Admin Rule tells you to ignore a specific topic/folder/company, or to ONLY use a specific folder/file, you MUST follow it strictly. If the 'Context' provided below is empty or doesn't match the required folder, you MUST state that you don't have the information in that specific folder.z{chat_history}z[Prior history provided below]z {context}z[Context provided below]z {question}z[Question provided below]z# ### GENERAL OPERATING PRINCIPLES zAdmin Override Detected: z itemsz.No active prompts found. Using default prompt.zError fetching prompts: )timerrrfindsortto_listsrc.utils.promptsrgetrDrUrnjoinrloweranyreplacerrArsrt)rJrcursoractive_promptsrpcontentsr<r custom_headerc custom_blockstrict_enforcementcleaned_default full_promptrrs @r_get_system_instructionsz,RERAInferenceEngine._get_system_instructionss   499;1H1H#H3#N%% %6 )WWY',,k4-@AFF|UVWF#)>>>#==N ? ?;I]QAEER[L\AEE)R0]]KK$$vc(m_D&26D/   !QR%:".2+&*iikD #!!!k>^>ZB ) KK   8< = ?*.D '( (  )sn=J AIH;IH>5H> AI$ I 0C?I/ J ;I> I J,J=J>J JJ system_instructionscf|jr|j|k(r |jStj|}|jj ddi}d}d|zt |zt dt dd|z|jztz}||_||_|S) Nkr, search_kwargsc dj|Dcgc].}d|jjddd|j0c}Scc}w)N SOURCE: rbUnknownr>)rr1rr0)rds r format_docsz3RERAInferenceEngine._get_chain..format_docs-sF;;kopfg(1::>>(9+M*NbQRQ_Q_P` apq qps3A c |dSNquestionr?xs rz0RERAInferenceEngine._get_chain..2s a mrc |dSrr?rs rrz0RERAInferenceEngine._get_chain..3s Qz]rc&|jddS)N chat_historyNo prior history.)rrs rrz0RERAInferenceEngine._get_chain..4s~Ob9cr)contextrr) rrr from_templater as_retrieverrrr)rJrprompt retrieverr new_chains r _get_chainzRERAInferenceEngine._get_chain$s   $"8"8R]C^^*+BC ./c d     hh    '!4rqueryrrc  Kddl}|j} |jj}|j}|jd{}|j}|jj d||z dd|jj d|ddd|j} |j jdd i } | j|d{} |jr>|jj d |jt| |j} ddl } | jd || j} | r| jd jnd}g}gd}|D]D}| j||}|s|j!|Dcgc]}|jc}Ft#t%|Dcgc] }|dvs| c}}g}gd}|D]D}| j||}|s|j!|Dcgc]}|jc}Ft#t%|Dcgc] }|dvs| c}}|j} ddi}|j j| } | j|d{} g}| D]}|j&j)dxs t+|dd}|j&j)dxs t+|dd}t-|j}t-|j}d}|rd}|D]}||k(s|s ||vsd}n|r|r|D]}||k(s|s ||vsd}n|r|j/||jj d|d|d|dd }dj1|Dcgc].}d|j&j)ddd|j20c}} |jj d|j| z dd t5|d!t5| d|j}!d"|fd#d$| d%|d&|fg}"|j6j|"d{}#|#j8}$|j}%|jj d'|%|!z dd|j}&|$j;d(dj;d)dj;dd*j;dd*j}$d+|$vr|$j;d+d*}$d+|$vr|jj d,|j|&z dd|jj d-|j|z dd|$S7Z7cc}wcc}wcc}wcc}w7cc}w7E#t<$r%}'|jj?d.|'|'d}'~'wwxYww)/NrzPrompt fetch took z.4fszSYSTEM_PROMPT_USED: r^z...rr,rz Applying admin override filter: zu### ABSOLUTE PRIORITY: CUSTOM ADMIN RULES \(OVERRIDE ALL OTHER RULES\)\n(.*?)(?=\n### GENERAL OPERATING PRINCIPLES|$)rZr)z.(?:from|in|use|of)\s+([a-zA-Z0-9_-]+)\s+folderz ([a-zA-Z0-9_-]+)\s+folder\s+onlyz([a-zA-Z0-9_-]+)\s+file\s+onlyz'only\s+answer\s+from\s+([a-zA-Z0-9_-]+))themythisaan)z"ignore\s+([a-zA-Z0-9_-]+)\s+folderz ignore\s+([a-zA-Z0-9_-]+)\s+filez)(?:don't|do not)\s+use\s+([a-zA-Z0-9_-]+)2rbrTFzPost-filter REJECTED: z (Folder: )rrrr>zRetrieval took z.2fz s (Result: z docs after filtering rdhumanz Context: z History: z Question: zGeneration took z*** z zCleaning took zTotal execute_query delay: zEngine execution error: ) rrqrrrDrUrrainvokerr:rrDOTALLrfindallextendrlsetr1rgetattrr3r4rr0rnrr8rrsrt)(rJrrr t_total_start norm_queryt_promptr t_prompt_endt_retrrradmin_constraints_matchadmin_constraintsallowed_folders patterns_onlypatmatchesmfignored_folderspatterns_ignorerr6r source_val folder_val source_lower doc_folder is_allowed final_docsrt_genmessages raw_responseresponse t_gen_endt_cleanrs( r execute_queryz!RERAInferenceEngine.execute_query?s- K ,,.Jyy{H(,(E(E(G"G 99;L KK  1,2I#1NaP Q KK  34G4M3NcR SIIKE))66c1X6NI"**511D**   #CDD_D_C`!ab0t7R7RS &(ii1ik~@B@I@I'J #Lc 7 = =a @ F F Hik  OM % I**S*;<#**w+G!AGGI+GH I #3?'paaOoFo'p#qrO OO ' I**S*;<#**w+G!AGGI+GH I #3?'paaOoFo'p#qrOIIKE !"IM))66]6SI"**511DM eZZ^^H5QHb9Q ZZ^^M:[gaXZ>[ ":446  _224 ! "!&J," ?qQ,5F)-J! " /," ?qQ,5F).J!" !((+KK$$'=l^:V`Uaab%cd5 e:'r*Jkknx"yijXajjnnXi.P-QQSTUTbTbSc#d"yzG KK  tyy{U/B3.G{SVWaSbRccyz}C{DzEEFG HIIKE./Jwi~l^>Z_Y`abH"&!1!1(!;;L#++H I KK  / E0A#/FaH IiikG''b199#rBJJ6SVW__`dfijpprH("#++D#6(" KK  ~diikG.CC-HJ K KK  :499;;VWZ:[[\] ^Oy#H2,,H(q,H(q2D#z<*  KK   8< =G sX AWV3BW5V66B1W(W7V9 W% V> /V> 3*WW-W W W %W )AW8W 9B#WW"W8W=AW3WBWWB=WA#W2X 3W6W9WW X! XXX cd|_d|_i|_d|_d|_|j j dy)z Invalidates all internal caches.NrzInference Engine cache cleared.)rrrrrrDrUrJs r clear_cachezRERAInferenceEngine.clear_caches=!"#!!!% :;rNr) rrrr rrrKrr3rrrr?rrrrsQ+%=+w~~+$>"@c6NNCNZ]N`rKs ,,W6-C5H92$811 1 "DN"(3-"TXYaTb"H7#7Xc]7X[7eh7$}}~E<E