Please enable JavaScript.
Coggle requires JavaScript to display documents.
Dynamic data lookup, 1, 2, 3, 4, 5, Step1, Step2, Step3, Step4, 6, 7, ALF …
Dynamic data lookup
you control
Which lookup datasets are loaded and how many copies of each
When a lookup dataset is loaded and unloaded. A lookup dataset is loaded when you call lookup_load on it within a transform. If you do not explicitly unload a dataset by later calling lookup_unload on it, the dataset is unloaded when its parent component process ends.
-
Lookup templates and dynamic loading
Dynamic loading is enabled by lookup templates, implemented as either a LOOKUP TEMPLATE component or a lookup template type
-
A lookup template type
In your transform or inline DML, create a lookup template type — a user-designed record of a certain format. This approach frees dynamic data lookup from requiring any lookup components at all and is implemented entirely in DML. You then create and use one or more instances, or handles, of this type to perform dynamic lookup operations
The type is either declared on the fly within the inline DML, or is included by the inline DML from a package file on disk through the AB_DML_DEFS graph parameter.
-
Load the lookup file using the lookup_load function inside a transform function
let my_lookup_type my_lookup_handle = lookup_load(MyData, MyIndex);
// datapath arg omitted
let my_lookup_type my_lookup_handle = lookup_load(MyIndex);
// datapath, indexpath args both omitted
let my_lookup_type my_lookup_handle = lookup_load();
lookup(my_lookup_handle, Expression);
Expression - Key fields i.e in.firstname
-
Characteristics of a lookup data file
Lookup data file path
Lookup index file path, if a precomputed index is present
Record format of the lookup data
Key by which the data records are sorted and to be searched
out::reformat(in) =
begin
let lookup_identifier_type LID = lookup_load(datafile, indexfile, "MyLookupTemplate");
out :: lookup(LID, "MyLookupTemplate", in.key);
end;
Dynamic lookup methods
LOOKUP_LOAD_ALL_GENERATIONS : 0
LOOKUP_LOAD_ALL_COMPLETE_GENERATIONS : -1
LOOKUP_LOAD_APPENDABLE : -2
LOOKUP_LOAD_UPDATABLE : -3
LOOKUP_LOAD_NO_GENERATIONS : -4
LOOKUP_LOAD_LAZY_APPENDABLE : -5
Appendable lookup files (ALFs) a newly arriving record is made available to your graph as soon as the complete record appears on disk. ALFs can enable your applications to process new data quickly — often less than a second after it is landed to disk.
Restrictions
You load only ordinary (exact matching, non-compressed) lookup files into memory
ou do not specify any index file during loading — indexing is performed on the fly as needed and is not precomputed.
You load lookup data that follows the record format specified in the lookup template
Syntax
let lookup_identifier_type LID = lookup_load($DATA/mydata, NULL, "My_Template", -2)
-
Updatable Lookup file
An updatable lookup file is a special kind of dynamically loaded lookup file that you build within a single graph component — often within a single transform, and reuse in a later phase or component.
Begins with an existing lookup file on disk, either empty or containing data
Index file path - NULL
Steps
Load that file into memory by calling the lookup_load function
For the load_behavior argument, the integer -3.
For the indexpath argument, the DML constant NULL
-
Limitation
In updatable mode, only one copy of a lookup data file can be loaded in memory at a given time.
-
Summary of lookup applications, types, sizes, and operations
Refer this from help file for detailed overview and differences
-
-
-
-
-
-
-
-
-
-
-
ALF - one process can write the data and one or more other processes can read the data from the same file asynchronously
Updatable lookup file - A single process needs to both read and write the data
-
-
-
