#include "common.h"

#ifdef INFORMATION

This file covers routines that createand access a "dictionary entry" (WORDP) and the "meaning" of words(MEANING).

The dictionary acts as the central hash mechanism for accessing various kinds of data.

The dictionary consists of data imported from WORDNET 3.0 (copyright notice at end of file) + augmentations + system and script pseudo - words.

A word also gets the WordNet meaning ontology(->meanings & ->meaningCount).The definition of meaning in WordNet

is words that are synonyms in some particular context.Such a collection in WordNet is called a synset.

Since words can have multiple meanings(and be multiple parts of speech), the flags of a word are a summary

of all of the properties it might haveand it has a list of entries called "meanings".Each entry is a MEANING

and points to the circular list, one of which marks the word you land at as the synset head.

This is referred to as the "master" meaning and has the gloss(definition) of the meaning.The meaning list of a master node points back to

all the real words which comprise it.

Since WordNet has an ontology, its synsets are hooked to other synsets in various relations, particular that

of parentand child.ChatScript represents these as facts.The hierarchy relation uses the verb "is" and

has the child as subject and the parent as object.Such a fact runs from the master entries, not any of the actual

word entries.So to see if "dog" is an "animal", you could walk every meaning list of the word animaland

mark the master nodes they point at.Then you would search every meaning of dog, jumping to the master nodes,

then look at facts with the master node as subjectand the verb "is" and walk up to the object.If the object is

marked, you are there.Otherwise you take that object node as subjectand continue walking up.Eventually you arrive at

a marked node or run out at the top of the tree.

Some words DO NOT have a master node.Their meaning is defined to be themselves(things like pronouns or determiners), so

their meaning value for a meaning merely points to themselves.

The meaning system is established by building the dictionaryand NEVER changes thereafter by chatbot execution.

New words can transiently enter the dictionary for various purposes, but they will not have "meanings".

A MEANING is a reference to a specific meaning of a specific word.It is an index into the dictionary

(identifying the word) and an index into that words meaning list(identifying the specific meaning).

An meaning list index of 0 refers to all meanings of the word.A meaning index of 0 can also be type restricted

so that it only refers to noun, verb, adjective, or adverb meanings of the word.

Since there are only two words in WordNet with more than 63 meanings(breakand cut) we limit all words to having no

more than 63 meanings by discarding the excess meanings.Since meanings are stored most important first,

these are no big loss.This leaves room for the 5 essential type flags used for restricting a generic meaning.

Space for dictionary words comes from a common pool.Dictionary words are

allocated linearly forward in the pool.Strings have their own pool(the heap).

All dictionary entries are indexable as a giant array.

The dictionary separately stores uppercaseand lowercase forms of the same words(since they have different meanings).

There is only one uppercase form stored, so Unitedand UnItED would be saved as one entry.The system will have

to decide which case a user intended, since they may not have bothered to capitalize a proper noun, or they

may have shouted a lowercase noun, and a noun at the start of the sentence could be either proper or not.

Dictionary words are hashed as lower case, but if the word has an upper case letter it will be stored

in the adjacent higher bucket.Words of the basic system are stored in their appropriate hash bucket.

After the basic system is read in, the dictionary is frozen.This means it remembers the spots the allocation

pointers are at for the dictionaryand heap spaceand is using mark - release memory management.

The hash buckets are themselves dictionary entries, sharing space.After the basic layers are loaded,

new dictionary entries are always allocated.Until then, if the word hashs to an empty bucket, that bucket

becomes the dictionary entry being added.

We mark sysnet entries with the word& meaning number& POS of word in the dictionary entry.The POS not used explicitly by lots of the system

but is needed when seeing the dictionary definitions(:word) and if one wants to use pos - restricted meanings in a match or in keywords.

#endif

int worstDictAvail = 1000000;

bool dictionaryBitsChanged = false;

HEAPREF propertyRedefines = NULL; // property changes on locked dictionary entries

HEAPREF flagsRedefines = NULL; // systemflags changes on locked dictionary entries

HEAPREF ongoingDictChanges = NULL; // ability to revert dynamic changes

bool monitorDictChanges = false;

bool xbuildDictionary = false; // indicate when building a dictionary

char dictionaryTimeStamp[20]; // indicate when dictionary was built

const char* mini = ""; // what language

unsigned int* hashbuckets = 0;

unsigned int wordAccessCount = 0; // number of words looked up in this volley

unsigned int bucketDepthCount = 0; // sum of depths required for each word lookup

static unsigned char* writePtr; // used for binary dictionary writes

// memory data

unsigned long maxHashBuckets = MAX_HASH_BUCKETS;

bool setMaxHashBuckets = false;

uint64 maxDictEntries = MAX_DICTIONARY;

MEANING posMeanings[64]; // concept associated with propertyFlags of WORDs

MEANING sysMeanings[64]; // concept associated with systemFlags of WORDs

#include <map>

using namespace std;

std::map <WORDP, WORDP> irregularNouns;

std::map <WORDP, WORDP> irregularVerbs;

std::map <WORDP, WORDP> irregularAdjectives;

std::map <WORDP, WORDP> canonicalWords;

std::map <WORDP, int> wordValues; // per volley

std::map <WORDP, MEANING> backtracks; // per volley

std::map <WORDP, int> countData;

HEAPREF concepts[MAX_SENTENCE_LENGTH]; // concept chains per word

HEAPREF topics[MAX_SENTENCE_LENGTH]; // topics chains per word

bool fullDictionary = true; // we have a big master dictionary, not a mini dictionary

bool primaryLookupSucceeded = false;

void LoadRawDictionary(int mini);

bool TraceHierarchyTest(int x)

{

if (!(x & TRACE_HIERARCHY)) return false;

x &= -1 ^ (TRACE_ON | TRACE_ALWAYS | TRACE_HIERARCHY | TRACE_ECHO);

return x == 0; // must exactly have been trace hierarchy

}

MEANING GetMeaning(WORDP D, int index)

{

MEANING* meanings = GetMeanings(D);

if (meanings) return meanings[index];

else return MakeMeaning(D); // switch to generic non null meaning

}

void SetTried(WORDP D, int value)

{

triedData[D] = value;

}

void RemoveConceptTopic(HEAPREF list[256], WORDP D, int index)

{

MEANING M = MakeMeaning(D);

HEAPREF at = list[index];

HEAPREF prior = NULL;

while (at)

{

uint64 m;

uint64* currentEntry = (uint64*)at;

at = UnpackHeapval(at, m, discard);

if ((uint64)M == m)

{

if (prior == NULL) list[index] = at; // list head

else ((uint64*)prior)[0] = currentEntry[0]; // splice this out

return; // assumed only on list once

}

prior = at;

}

}

void Add2ConceptTopicList(HEAPREF list[256], WORDP D, int start, int end, bool unique)

{

int index = Word2Index(D);

if (unique)

{

HEAPREF at = list[start];

while (at)

{

uint64 D1;

at = UnpackHeapval(at, D1, discard, discard);

if (D1 == (uint64)index) return; // already on list

}

}

// concepts[] and topics[]

list[start] = AllocateHeapval(list[start], (uint64)index, 0, 0);

}

void ClearHeapThreads()

{

memoryMarkThreadList = NULL;

userVariableThreadList = NULL;

savedSentencesThreadList = NULL;

propertyRedefines = NULL; // property changes on locked dictionary entries

flagsRedefines = NULL; // system flag changes on locked dictionary entries

factThread = NULL;

rulematches = NULL;

patternwordthread = NULL;

variableChangedThreadlist = NULL;

memoryVariableThreadList = NULL; // list of variable changes for each mark

memoryVariableChangesThreadList = NULL; // list of original values in this mark

for (int i = 1; i < MAX_SENTENCE_LENGTH; ++i) concepts[i] = topics[i] = NULL;

// kernelVariableThreadList

// botVariableThreadList;

}

void ClearVolleyWordMaps()

{

wordValues.clear();

backtracks.clear();

ClearWhereInSentence();

ClearTriedData(); // prevent document reuse

}

void ClearWordMaps() // both static for whole dictionary and dynamic per volley

{

irregularNouns.clear();

irregularVerbs.clear();

irregularAdjectives.clear();

canonicalWords.clear();

triedData.clear(); // prevent document reuse

ClearVolleyWordMaps();

}

void ClearWhereAt(int where) // remove all concepts and markings at this slot in sentence

{

WORDP D;

concepts[where] = NULL; // drop memory list at this slot

topics[where] = NULL; // drop memory list

WORDP holdlist[10000];

int holdint[10000];

int mark[10000];

memset(mark, 0, sizeof(mark));

unsigned int index = 0;

bool changed = false;

// copy because we will rewrite triedData.

map<WORDP, int>::iterator it1;

for (it1 = triedData.begin(); it1 != triedData.end(); ++it1)

{

D = it1->first;

int x = it1->second;

holdint[index] = x;

holdlist[index++] = D;

}

for (unsigned int item = 0; item < index; ++item)

{

int access = holdint[item]; // heap access

char* data = Index2Heap(access);

if (data == 0) continue;

unsigned char* lcltriedData = (unsigned char*)(data + 8); // skip over 64bit tried by meaning field

int i;

int counter = 0;

for (i = 0; i < MAXREFSENTENCE_BYTES; i += REF_ELEMENT_SIZE) // 6 bytes per entry x 300 entries

{

unsigned char start = lcltriedData[i];

if (start == where) // modify it

{

if (!mark[item]) // change over to a copy (dont harm original)

{

int newaccess = CopyWhereInSentence(access);

holdint[item] = newaccess;

mark[item] = 1;

changed = true;

data = Index2Heap(newaccess);

lcltriedData = (unsigned char*)(data + 8); // skip over 64bit tried by meaning field

}

int remain = MAXREFSENTENCE_BYTES - i - REF_ELEMENT_SIZE;

if (remain == 0) // just chop it off

{

lcltriedData[i] = 0xff;

lcltriedData[i + 1] = 0xff;

break;

}

memmove(lcltriedData + i, lcltriedData + i + REF_ELEMENT_SIZE, remain); // remove element

// insert end marker at end

lcltriedData[MAXREFSENTENCE_BYTES - REF_ELEMENT_SIZE] = 0xff;

lcltriedData[MAXREFSENTENCE_BYTES - REF_ELEMENT_SIZE + 1] = 0xff;

if (++counter > 100) // precaution

break;

i -= REF_ELEMENT_SIZE;

continue; // check next one

}

else if (start > where) break;

}

}

if (changed)

{

for (unsigned int item = 0; item < index; ++item)

{

if (!mark[item]) continue;

D = holdlist[item];

int access = holdint[item]; // heap access

triedData[D] = access; // replace now

}

}

}

void ClearWordWhere(WORDP D, int at)

{

triedData.erase(D);

// also remove from concepts/topics lists

if (at == -1)

{

for (unsigned int i = 1; i <= (unsigned int)wordCount; ++i)

{

RemoveConceptTopic(concepts, D, i);

RemoveConceptTopic(topics, D, i);

}

}

}

void SetFactBack(WORDP D, MEANING M)

{

if (GetFactBack(D) == 0)

{

backtracks[D] = M;

}

}

MEANING GetFactBack(WORDP D)

{

std::map<WORDP, MEANING>::iterator it;

it = backtracks.find(D);

return (it != backtracks.end()) ? it->second : 0;

}

void ClearBacktracks()

{

backtracks.clear();

}

void SetPlural(WORDP D, MEANING M)

{

irregularNouns[D] = dictionaryBase + M; // must directly assign since word on load may not exist

}

void SetComparison(WORDP D, MEANING M)

{

irregularAdjectives[D] = dictionaryBase + M; // must directly assign since word on load may not exist

}

void SetTense(WORDP D, MEANING M)

{

irregularVerbs[D] = dictionaryBase + M; // must directly assign since word on load may not exist

}

void SetCanonical(WORDP D, MEANING M)

{

canonicalWords[D] = dictionaryBase + M; // must directly assign since word on load may not exist

}

WORDP RawCanonical(WORDP D)

{

std::map<WORDP, WORDP>::iterator it;

it = canonicalWords.find(D);

if (it == canonicalWords.end()) return NULL;

return it->second;

}

WORDP GetCanonical(WORDP D)

{

std::map<WORDP, WORDP>::iterator it;

it = canonicalWords.find(D);

if (it == canonicalWords.end()) return NULL;

WORDP E = it->second;

if (*E->word != '`') return E; // normal english canonical

// foreign canonicals have multiple choices. This code only picks first one. Other code may decide more complex, but generally treetagger will do that

char word[MAX_WORD_SIZE];

strcpy(word, E->word + 1);

char* end = strchr(word, '`');

*end = 0;

end = strchr(word, '|'); // multiple choice?

if (end) *end = 0; // pick 1st one

return StoreWord(word, AS_IS);

}

WORDP GetTense(WORDP D)

{

std::map<WORDP, WORDP>::iterator it;

it = irregularVerbs.find(D);

return (it != irregularVerbs.end()) ? it->second : NULL;

}

WORDP GetPlural(WORDP D)

{

std::map<WORDP, WORDP>::iterator it;

it = irregularNouns.find(D);

return (it != irregularNouns.end()) ? it->second : NULL;

}

WORDP GetComparison(WORDP D)

{

std::map<WORDP, WORDP>::iterator it;

it = irregularAdjectives.find(D);

return (it != irregularAdjectives.end()) ? it->second : NULL;

}

void SetWordValue(WORDP D, int x)

{

wordValues[D] = x;

}

int GetWordValue(WORDP D)

{

std::map<WORDP, int>::iterator it;

it = wordValues.find(D);

return (it != wordValues.end()) ? it->second : 0;

}

// start and ends of space allocations

WORDP dictionaryBase = NULL; // base of allocated space that encompasses dictionary, heap space, and meanings

WORDP dictionaryFree = NULL; // current next dict space available going forward (not a valid entry)

// return-to values for layers

WORDP dictionaryPreBuild[NUMBER_OF_LAYERS + 1];

char* heapPreBuild[NUMBER_OF_LAYERS + 1];

// build0Facts

// return-to values after build1 loaded, before user is loaded

WORDP dictionaryLocked = NULL;

FACT* factLocked = 0;

char* stringLocked;

// format of word looked up

uint64 verbFormat;

uint64 nounFormat;

uint64 adjectiveFormat;

uint64 adverbFormat;

// dictionary ptrs for these words

WORDP Dplacenumber;

WORDP Dpropername;

MEANING Mphrase;

MEANING MabsolutePhrase;

MEANING MtimePhrase;

WORDP Dclause;

WORDP Dverbal;

WORDP Dmalename, Dfemalename, Dhumanname;

WORDP Dtime;

WORDP Dunknown;

WORDP Dchild, Dadult;

WORDP Dtopic;

MEANING Mchatoutput;

MEANING Mburst;

MEANING Mpending;

MEANING Mkeywordtopics;

MEANING Mconceptlist;

MEANING Mmoney;

MEANING Mintersect;

MEANING MgambitTopics;

MEANING MadjectiveNoun;

MEANING Mnumber;

WORDP Dpronoun;

WORDP Dadjective;

WORDP Dauxverb;

WORDP DunknownWord;

static char* predefinedSets[] = // some internally mapped concepts not including emotions from LIVEDATA/interjections

{

(char*)"~repeatme",(char*)"~repeatinput1",(char*)"~repeatinput2",(char*)"~repeatinput3",(char*)"~repeatinput4",(char*)"~repeatinput5",(char*)"~repeatinput6",(char*)"~uppercase",(char*)"~utf8",(char*)"~sentenceend",

(char*)"~pos",(char*)"~sys",(char*)"~grammar_role",(char*)"~daynumber",(char*)"~yearnumber",(char*)"~dateinfo",(char*)"~formatteddate",(char*)"~email_url",(char*)"~fahrenheit",(char*)"~celsius",(char*)"~kelvin",

(char*)"~kindergarten",(char*)"~grade1_2",(char*)"~grade3_4",(char*)"~grade5_6",(char*)"~twitter_name",(char*)"~hashtag_label",(char*)"~filename",

(char*)"~shout",(char*)"~distance_noun_modify_adverb",

(char*)"~distance_noun_modify_adjective",(char*)"~modelnumber",

(char*)"~passive_verb",

(char*)"~time_noun_modify_adverb",

(char*)"~time_noun_modify_adjective",

(char*)"~capacronym",(char*)"~emoji",

NULL

};

void RestorePropAndSystem(char* stringUsed)

{

uint64 D;

while (propertyRedefines) // must release these

{

uint64 properties;

if (stringUsed && propertyRedefines >= stringUsed) break; // not part of this freeing

propertyRedefines = UnpackHeapval(propertyRedefines, D, properties, discard);

((WORDP)D)->properties = properties;

}

while (flagsRedefines) // must release these

{

uint64 properties;

if (stringUsed && flagsRedefines >= stringUsed) break; // not part of this freeing

flagsRedefines = UnpackHeapval(flagsRedefines, D, properties, discard);

((WORDP)D)->systemFlags = properties;

}

}

void DictionaryRelease(WORDP until, char* stringUsed)

{

RestorePropAndSystem(stringUsed);

ongoingDictChanges = NULL; // discard user tracked changes since we have our own essential tracking

if (until) while (dictionaryFree > until) DeleteDictionaryEntry(--dictionaryFree); // remove entry from buckets

lastheapfree = heapFree = stringUsed;

}

char* UseDictionaryFile(const char* name)

{

static char junk[100];

if (*mini) sprintf(junk, (char*)"DICT/%s", mini);

else if (!*language) sprintf(junk, (char*)"%s", (char*)"DICT");

else if (!name) sprintf(junk, (char*)"DICT/%s", language);

else sprintf(junk, (char*)"DICT/%s", language);

MakeDirectory(junk); // if it doesnt exist

if (name && *name)

{

strcat(junk, (char*)"/");

strcat(junk, name);

}

return junk;

}

MEANING FindChild(MEANING who, int n)

{ // GIVEN SYNSET

FACT* F = GetObjectNondeadHead(who);

unsigned int index = Meaning2Index(who);

while (F)

{

FACT* at = F;

F = GetObjectNondeadNext(F);

if (at->verb != Mis) continue;

if (index && at->object != who) continue; // not us

if (--n == 0) return at->subject;

}

return 0;

}

bool ReadForeignPosTags(const char* fname)

{

FILE* in = FopenReadOnly(fname);

if (!in) return false;

char word[MAX_WORD_SIZE];

char wordx[MAX_WORD_SIZE];

*word = '_'; // marker to keep any collision away from foreign pos

*wordx = '~'; // corresponding concept

while (ReadALine(readBuffer, in) >= 0) // foreign name followed by bits of english pos

{

char* ptr = ReadCompiledWord(readBuffer, word + 1);

if (!word[1] || word[1] == '#')

continue;

uint64 flags = 0;

char flag[MAX_WORD_SIZE];

while (*ptr) // get english pos values of this tag

{

ptr = ReadCompiledWord(ptr, flag);

if (!*flag || *flag == '#') break;

uint64 val = FindPropertyValueByName(flag);

if (!val)

{

(*printer)("Unable to find flag %s\r\n", flag);

}

flags |= val;

}

StoreWord(word, flags); // _foreignpos gets bits

MakeLowerCopy(wordx + 1, word + 1); // force the name to lowercase because the name might not be valid, e.g. DET:ART

BUILDCONCEPT(wordx);

}

fclose(in);

// Also store the unknown tag, just in case the pos tagger fails to attach anything

strcpy(word + 1, (char*)"unknown-tag");

StoreWord(word, 0);

*word = '~'; // corresponding concept

BUILDCONCEPT(word);

return true;

}

unsigned char BitCount(uint64 n)

{

unsigned char count = 0;

while (n)

{

count++;

n &= (n - 1);

}

return count;

}

WORDP GetSubstitute(WORDP D)

{

return (D && D->systemFlags & HAS_SUBSTITUTE) ? D->w.substitutes : 0;

}

void BuildShortDictionaryBase();

static void EraseFile(const char* file)

{

FILE* out = FopenUTF8Write(UseDictionaryFile(file));

FClose(out);

}

void ClearDictionaryFiles()

{

char buffer[MAX_WORD_SIZE];

remove(UseDictionaryFile((char*)"dict.bin"));

remove(UseDictionaryFile((char*)"facts.bin"));

EraseFile((char*)"dict.bin"); // create but empty file

EraseFile((char*)"facts.bin"); // create but empty file

unsigned int i;

for (i = 'a'; i <= 'z'; ++i)

{

sprintf(buffer, (char*)"%c.txt", i);

EraseFile(buffer); // create but empty file

}

for (i = '0'; i <= '9'; ++i)

{

sprintf(buffer, (char*)"%c.txt", i);

EraseFile(buffer); // create but empty file

}

}

void BuildDictionary(char* label)

{

xbuildDictionary = true;

int miniDict = 0;

char word[MAX_WORD_SIZE];

mini = language;

char* ptr = ReadCompiledWord(label, word);

bool makeBaseList = false;

if (!stricmp(word, (char*)"wordnet")) // the FULL wordnet dictionary w/o synset removal

{

miniDict = -1;

ReadCompiledWord(ptr, word);

mini = "WORDNET";

}

else if (!stricmp(word, (char*)"basic"))

{

miniDict = 1;

makeBaseList = true;

FClose(FopenUTF8Write((char*)"RAWDICT/basicwordlist.txt"));

maxHashBuckets = 10000;

setMaxHashBuckets = true;

mini = "BASIC";

}

else if (!stricmp(word, (char*)"layer0") || !stricmp(word, (char*)"layer1")) // a mini dictionary

{

miniDict = !stricmp(word, (char*)"layer0") ? 2 : 3;

mini = (miniDict == 2) ? (char*)"LAYER0" : (char*)"LAYER1";

maxHashBuckets = 10000;

setMaxHashBuckets = true;

}

else if (stricmp(language, (char*)"english")) // a foreign dictionary

{

miniDict = 6;

maxHashBuckets = 10000;

setMaxHashBuckets = true;

}

UseDictionaryFile(NULL);

PartiallyCloseSystem();

InitStackHeap();

InitFacts(); // must do facts before dict since dict defines system facts

InitTextUtilities(); // also part of basic system before a build

InitDictionary();

InitTextUtilities1(); // also part of basic system before a build

InitUserCache();

LoadRawDictionary(miniDict);

if (miniDict && miniDict != 6) StoreWord((char*)"minidict"); // mark it as a mini dictionary

// dictionary has been built now

(*printer)((char*)"%s", (char*)"Dumping dictionary\r\n");

ClearDictionaryFiles();

WalkDictionary(WriteDictionary);

if (makeBaseList) BuildShortDictionaryBase(); // write out the basic dictionary

remove(UseDictionaryFile((char*)"dict.bin")); // invalidate cache of dictionary, forcing binary rebuild later

myBot = 0;

WriteFacts(FopenUTF8Write(UseDictionaryFile((char*)"facts.txt")), factBase);

sprintf(logFilename, (char*)"%s/build_log.txt", usersfolder); // all data logged here by default

FILE* out = FopenUTF8Write(logFilename);

FClose(out);

(*printer)((char*)"dictionary dump complete %d\r\n", miniDict);

echo = true;

xbuildDictionary = false;

CreateSystem();

}

void InitDictionary()

{

// read what the default dictionary wants as hash if parameter didnt set it

if (!setMaxHashBuckets)

{

FILE* in = FopenStaticReadOnly(UseDictionaryFile((char*)"dict.bin")); // DICT

if (in)

{

unsigned int check = Read32(in); // checkstamp

if (check == CHECKSTAMP) maxHashBuckets = Read32(in); // bucket size used by dictionary file

FClose(in);

}

}

dictionaryLocked = 0;

size_t size = (size_t)(sizeof(WORDENTRY) * maxDictEntries);

size /= sizeof(WORDENTRY);

size = (size * sizeof(WORDENTRY)) + sizeof(WORDENTRY);

size /= 64;

size = (size * 64) + 64;

int hsize = (maxHashBuckets + 1) * sizeof(int);

if (!dictionaryBase)

{

// dictionary and meanings

// on FUTURE startups (not 1st) the userCacheCount has been preserved while the rest of the system is reloaded

hashbuckets = (unsigned int*)malloc(size + hsize);

if (!hashbuckets) ReportBug("FATAL: Cannot allocate dictionary space");

dictionaryBase = (WORDP)(((char*)hashbuckets) + hsize);

#ifdef EXPLAIN

Conjoined heap space Independent heap space

heapBase -- allocates downwards

heapBase - allocates downwards heapFree

heapFree heapEnd -- stackFree - allocates upwards

------------------------ - disjoint memory

dictionaryFree

dictionaryBase - allocates upwards

userTopicStore 1..n

userTable refers to userTopicStore

cacheBase

#endif

}

memset(hashbuckets, 0, size + hsize);

dictionaryFree = dictionaryBase + 1; // dont write on 0

dictionaryPreBuild[LAYER_0] = dictionaryPreBuild[LAYER_1] = dictionaryPreBuild[LAYER_BOOT] = 0; // in initial dictionary

factsPreBuild[LAYER_0] = factsPreBuild[LAYER_1] = factsPreBuild[LAYER_BOOT] = lastFactUsed; // last fact in dictionary

}

void AddInternalFlag(WORDP D, unsigned int flag)

{

if (flag && (flag & D->internalBits) != flag) // prove there is a change - dont use & because if some bits are set is not enough

{

if (D < dictionaryLocked && !csapicall) // api can change substitutions

return;

D->internalBits |= (unsigned int)flag;

if (monitorChange) D->internalBits |= BIT_CHANGED;

}

}

void RemoveInternalFlag(WORDP D, unsigned int flag)

{

D->internalBits &= -1 ^ flag;

}

static void PreserveSystemFlags(WORDP D)

{

flagsRedefines = AllocateHeapval(flagsRedefines, (uint64)D, D->systemFlags, 0);

}

void AddSystemFlag(WORDP D, uint64 flag)

{

if (flag & NOCONCEPTLIST && *D->word != '~') flag ^= NOCONCEPTLIST; // not allowed to mark anything but concepts with this

if (flag && flag != (D->systemFlags & flag)) // prove there is a change - dont use & because if some bits are set is not enough

{

if (D < dictionaryLocked) PreserveSystemFlags(D); // autoprotect old dictionary

if (monitorDictChanges) ongoingDictChanges = AllocateHeapval(ongoingDictChanges, (uint64)D, D->properties, D->systemFlags);

D->systemFlags |= flag;

if (monitorChange && flag != MARKED_WORD) D->internalBits |= BIT_CHANGED;

}

}

void AddParseBits(WORDP D, unsigned int flag)

{

if (flag && flag != (flag & D->parseBits)) // prove there is a change - dont use & because if some bits are set is not enough

{

if (D < dictionaryLocked)

return;

D->parseBits |= (unsigned int)flag;

if (monitorChange) D->internalBits |= BIT_CHANGED;

}

}

void RemoveSystemFlag(WORDP D, uint64 flags)

{

if (D->systemFlags & flags)

{

if (D < dictionaryLocked) PreserveSystemFlags(D);

D->systemFlags &= -1LL ^ flags;

}

}

static void PreserveProperty(WORDP D)

{

propertyRedefines = AllocateHeapval(propertyRedefines, (uint64)D, D->properties);

}

void ReverseDictionaryChanges(HEAPREF start)

{

uint64 val1, val2, val3;

while (start)

{

start = UnpackHeapval(start, val1, val2, val3);

if (val1) {

WORDP D = (WORDP)val1;

D->properties = val2;

D->systemFlags = val3;

}

}

}

void AddProperty(WORDP D, uint64 flag)

{

if (D && flag && flag != (D->properties & flag))

{

if (D < dictionaryLocked) PreserveProperty(D);

if (monitorDictChanges) ongoingDictChanges = AllocateHeapval(ongoingDictChanges, (uint64)D, D->properties, D->systemFlags);

if (!(D->properties & (PART_OF_SPEECH)) && flag & PART_OF_SPEECH && *D->word != '~' && *D->word != '^' && *D->word != USERVAR_PREFIX && flag != CURRENCY) // not topic,concept,function

{

// internal use, do not allow idioms on words from #defines or user variables or sets.. but allow substitutes to do it?

unsigned int n = BurstWord(D->word);

if (n != 1)

{

char* buffer = AllocateBuffer();

WORDP E = StoreWord(JoinWords(1, false, buffer), AS_IS); // create the 1-word header

if (n > GETMULTIWORDHEADER(E)) SETMULTIWORDHEADER(E, n); // mark it can go this far for an idiom

FreeBuffer();

}

}

D->properties |= flag;

if (monitorChange) D->internalBits |= BIT_CHANGED;

}

}

void RemoveProperty(WORDP D, uint64 flags)

{

if (D && D->properties & flags)

{

if (D < dictionaryLocked) PreserveProperty(D);

D->properties &= -1LL ^ flags;

}

}

bool StricmpUTF(char* w1, char* w2, int len)

{

unsigned char c1, c2;

unsigned char* word1 = (unsigned char*)w1;

unsigned char* word2 = (unsigned char*)w2;

while (len && *word1 && *word2)

{

if (*word1 == 0xc3) // utf8 case sensitive

{

c1 = *word1;

c2 = *word2;

if (c1 == c2);

else if (c1 >= 0x9f && c1 <= 0xbf) // lower case form

{

if (c2 >= 0x80 && c2 <= 0x9e && c1 != 0x9f && c1 != 0xb7) // uppercase form

{

c1 -= 0xa0;

c1 += 0x80;

}

if (c1 != c2) return true;

}

else if (c2 >= 0x9f && c2 <= 0xbf) // lower case form

{

if (c1 >= 0x80 && c1 <= 0x9e && c2 != 0x9f && c2 != 0xb7) // uppercase form

{

c2 -= 0xa0;

c2 += 0x80;

}

if (c1 != c2) return true;

}

}

else if (word1[1] >= 0xc4 && word1[1] <= 0xc9)

{

c1 = *word1;

c2 = *word2;

if (c1 == c2);

else if (c1 & 1 && (c1 & 0xf7) == c2); // lower case form

else if (c2 & 1 && (c2 & 0xf7) == c1);

else return true;

}

else if (toLowercaseData[*word1] != toLowercaseData[*word2]) return true;

int size = UTFCharSize((char*)word1);

word1 += size;

word2 += size;

len -= size;

}

if (len == 0) return false; // if matched all characters we needed to, then we are done

return *word1 || *word2;

}

int GetWords(char* word, WORDP * set, bool strictcase)

{

int index = 0;

size_t len = strlen(word);

if (len >= MAX_WORD_SIZE) return 0; // not legal

char commonword[MAX_WORD_SIZE];

strcpy(commonword, word);

char* at = commonword;

while ((at = strchr(at, ' '))) *at = '_'; // match as underscores whenever spaces occur (hash also treats them the same)

bool hasUpperCharacters;

bool hasUTF8Characters;

uint64 fullhash = Hashit((unsigned char*)word, len, hasUpperCharacters, hasUTF8Characters); // sets hasUpperCharacters and hasUTF8Characters

unsigned int hash = (fullhash % maxHashBuckets); // mod by the size of the table

// lowercase bucket

++wordAccessCount;

WORDP D = dictionaryBase + hashbuckets[hash];

char word1[MAX_WORD_SIZE];

if (strictcase && hasUpperCharacters) D = dictionaryBase; // lower case not allowed to match uppercase input

int limit = 100;

while (D != dictionaryBase && --limit)

{

++bucketDepthCount;

if (fullhash == D->hash && D->length == len)

{

strcpy(word1, D->word);

at = word1;

while ((at = strchr(at, ' '))) *at = '_'; // match as underscores whenever spaces occur (hash also treats them the same)

if (!StricmpUTF(word1, commonword, len)) set[index++] = D;

}

D = dictionaryBase + GETNEXTNODE(D);

}

// upper case bucket

++wordAccessCount;

D = dictionaryBase + hashbuckets[hash + 1];

if (strictcase && !hasUpperCharacters) D = dictionaryBase; // upper case not allowed to match lowercase input

int limitx = 100;

while (D != dictionaryBase && --limitx)

{

++bucketDepthCount;

if (fullhash == D->hash && D->length == len)

{

strcpy(word1, D->word);

at = word1;

while ((at = strchr(at, ' '))) *at = '_'; // match as underscores whenever spaces occur (hash also treats them the same)

if (!StricmpUTF(word1, commonword, len)) set[index++] = D;

}

D = dictionaryBase + GETNEXTNODE(D);

}

return index;

}

int UTFCharSize(char* utf)

{

unsigned int count; // bytes of character

unsigned char c = (unsigned char)*utf;

if (!(c & 0x80)) count = 1; // ordinary ascii char

else if ((c & 0xE0) == 0xC0) count = 2; // 110 0xc0

else if ((c & 0Xf0) == 0Xe0) count = 3; // 1110 0xE0

else if ((c & 0Xf8) == 0XF0) count = 4; // 11110 0xF0

else count = 1;

return count;

}

WORDP FindWord(const char* word, unsigned int len, uint64 caseAllowed)

{

if (word == NULL || *word == 0 || dictionaryBase == NULL) return NULL;

if (len == 0) len = strlen(word);

bool hasUpperCharacters;

bool hasUTF8Characters;

++wordAccessCount;

uint64 fullhash = Hashit((unsigned char*)word, len, hasUpperCharacters, hasUTF8Characters); // sets hasUpperCharacters and hasUTF8Characters

unsigned int hash = (fullhash % maxHashBuckets); // mod by the size of the table

if (caseAllowed & LOWERCASE_LOOKUP) { ; } // stay in lower bucket regardless

// rule label (topic.name) uses uppercase lookup

else if (*word == SYSVAR_PREFIX || *word == USERVAR_PREFIX || (*word == '~' && !strchr(word, '.')) || *word == '^')

{

if (caseAllowed == UPPERCASE_LOOKUP) return NULL; // not allowed to find

caseAllowed = LOWERCASE_LOOKUP; // these are always lower case

}

else if (hasUpperCharacters || (caseAllowed & UPPERCASE_LOOKUP)) ++hash;

// you can search on upper or lower specifically (not both) or primary or secondary or both

// normal or fixed case bucket

WORDP D;

unsigned int currentBucketDepthCount = bucketDepthCount;

primaryLookupSucceeded = true;

if (caseAllowed & (PRIMARY_CASE_ALLOWED | LOWERCASE_LOOKUP | UPPERCASE_LOOKUP))

{

D = Index2Word(hashbuckets[hash]);

WORDP almost = NULL;

WORDP preferred = NULL;

WORDP exact = NULL;

int limit = 100;

while (D != dictionaryBase && --limit)

{

++bucketDepthCount;

if (fullhash == D->hash && D->length == len && !StricmpUTF(D->word, (char*)word, len)) // they match independent of case-

{

if (caseAllowed == LOWERCASE_LOOKUP) return D; // incoming word MIGHT have uppercase letters but we will be in lower bucket

else if (hasUpperCharacters) // we are looking for uppercase or primary case and are in uppercase bucket

{

if (D->internalBits & PREFER_THIS_UPPERCASE) preferred = D;

else if (!strncmp(D->word, word, len)) exact = D; // exactly what we want in upper case

else almost = D; // remember semi-match in upper case

}

else // if uppercase lookup, we are in uppercase bucket (input was lower case) else we are in lowercase bucket

{

return D; // it is exactly what we want in lower case OR its an uppercase acceptable match

}

}

D = dictionaryBase + GETNEXTNODE(D);

}

if (preferred) return preferred;

else if (exact) return exact;

else if (almost) return almost; // uppercase request we found in a different form only

}

// alternate case bucket (checking opposite case)

bucketDepthCount = currentBucketDepthCount;

primaryLookupSucceeded = false;

if (caseAllowed & SECONDARY_CASE_ALLOWED)

{

WORDP almost = NULL;

WORDP preferred = NULL;

D = dictionaryBase + hashbuckets[hash + ((hasUpperCharacters) ? -1 : 1)];

int limit = 100;

while (D != dictionaryBase && --limit)

{

++bucketDepthCount;

if (fullhash == D->hash && D->length == len && !StricmpUTF(D->word, (char*)word, len))

{