Author's note: the following article was produced from the previous text-based version of the XFormDBF documentation and task list (xformdbf.txt) that accompanied the last program update on 3/18/00.  I transformed this text file mechanically into HTML with minimal editing to put it into a more useful, legible form.  There will be no further revisions of xformdbf.txt, as this task list has been superseded by the new XFormDBF Development Tasks page and its accompanying section in the table of contents, which is handled with a FrontPage include file for the outline of tasks.  The tasks described in these old notes will eventually be assimilated into the new system, but this will be a gradual process.

Notes About the XFormDBF Application
By M. Asherman, 3/18/2000

Intro:

The XFormDBF Application is a generic tool for transforming a DBF into a derivative DBF with user-defined mappings from the old to the new columns. XFormDBF uses the generic Saver mechanism to support state-saving and restoration, i.e. saved form configurations. This application is fully functional in a pure VFP runtime environment; it does not require installation of the VFP Development System.

To test it out, run this command from VFP:

? xformdbf()

This will launch the modal dialog, restoring its default configuration. Changes to this form are automatically saved and restored. See xformdbf.prg for an explanation of optional arguments.

Notes about error handling

XFormDBF returns a success flag plus additional detailed error information in a separate ErrHandler object, derived from one of the MDA foundation error-handling classes (see ...\appl\errhandl). The global memory variable m.xformerrh is used to hold on to this object reference, which the caller may pre-allocate. Otherwise xformdbf will automatically allocate an external ErrHandler and assign it to m.xformerrh. If m.xformerrh doesn't exist in the scope of xformdbf, XFormDBF allocates it as a PUBLIC memory variable.

Currently available documentation:

• brief descriptions have been attached to all files in XFormDBF.pjx
• note that certain generic mdacomm functions are employed in the meta-file
• all programs are liberally commented

Further outstanding issues (* indicates it can wait, % means needs discussion):

XFormDBF Documentation and Help issues

• rework the organization of this text file into a more manageable form
  The large number of detailed issues and task descriptions is pushing the limits of this crude, single file organization. Using HTML instead of text would allow the task list to be organized more conveniently. Other pieces really should be broken out into separate pages.
• support context-sensitive HTML Help
  VFP makes it easy to connect an application like xformdbf to context-sensitive HTML Help. This feature should be utilized as a way of providing the most convenient user documentation.
• incorporate developer-oriented documentation into help files
  Developer documentation can be regarded as an extension of the user documentation. This should also include links to related tasks and development planning documents.
• provide a convenient channel for feedback and discussion among users, developers
  Ideally, each piece of documentation or help should include a direct link to a related live discussion thread.

Misc. XFormDBF issues (other than optimization, listed separately below):

• supply configurable pushbutton(s) for convenient tie-ins to related functions
  (discussed with Alan on 2/29) Specifically, this could be used to invoke an application-specific lookup table maintenance facility. Similarly, we could use such a tie-in to get to the poor man's function library table browse.
• provide a configurable hook into app-specific error-handling logic
  E.g. let there be a user-definable expression, specified on the Options page, which would be evaluated during the xformdbf error-handling logic. This function would be responsible for logging errors and sorting out fatal from non-fatal cases. The function result could be used to indicate whether to disregard the error, etc.
• take a stab at minimal error-handling during state-restoration
  A complication is that we must continue to provide some means of either handling the special case of 1-time errors upon restoring old configurations, or provide another mechanism for dealing with version upgrades of Saver-based applications.
• support a flexible library of pre-defined mapping functions
  The general idea would be to support a combobox or listbox of pre-defined functions, whose definitions would reside in a Commander meta-file. It might be useful to support a flavor of "macro" functions, which immediately expand into a canned functional expression, as opposed to functions that are evaluated later. This will require further thought and clarification. As a preliminary approach, I suggested the following to Alan on 9/29: create a table of canned functional expressions, using a memo field to store the expression; browse this table while using xformdbf modelessly; voila - you've got a poor man's function library. With a few small xformdbf refinements, e.g. directly editable mapping expressions and a pushbutton tie-in to browse, this could be made still more convenient.
• optional toggle for absolute vs. relative path preferences
  (See email from Nadya and Nancy + mda reply of 10/21). In the meantime, this can be accomplished by passing explicit input/output file name arguments to xformdbf(), or by direct manipulation of the applicable meta-file records.
• may as well eliminate dev_mode behavior of enablement/disablement via rightclick
  Let dev mode for xformdbf make Options and Config tabs available, else conceal or disable these pages (or something along those lines).
• display current position and count of records in input and output files
• flaky widths of columns in the output grid - 10 char min default would be nice
  (Feature requested by Nadya on 10/19). Also note related task of saving grid column layouts, but there's also the simpler issue of fixing VFP's flaky column resizing behavior, where Restore Config causes column widths to shrink to too-narrow sizings.
• provide Display and/or Modify Structure button for the input table
  Requested by Nadya on 2/29.
• automatic form resizing logic would be nice
  Nadya brought this up again on 2/18. I agree, it would be nice, so I've raised the priority, but it still doesn't rise to the top. Since the default form size already fills most of the screen, it won't buy us that much more space.
• *- support a toggle to enable/disable display of mappings for missing fields
  (Feature requested by Nadya on 10/19).
• *- user-settable flag property to indicate whether SAFETY should be set ON or OFF for interactive cases. This could apply to both Clone and Zap commands. Note that SET SAFETY is scoped to the current data session, so it won't do to rely on the caller's setting.
• *- maybe allow the user to override edMapping validation errors
  to avoid hassles with obscure cases and incremental editing scenarios. Otherwise, the user could get stuck in the edMapping field, with no way easy to exit or override cleanly. Hold off on this for now, since the current validation logic seems reasonable in the vast majority of cases. The user can always force exit from validation by clearing the mapping and abandoning the config, if necessary. Wait and see if any additional escape mechanisms are really needed.
• *- maybe add support for a separate "zoom" window on editable mapping expression
  Not clear if this is really needed. Wait and see.
• *- maybe interactive Exit without running even once should be regarded as failure
  Wait and see if anyone cares.
• *- maybe move Test and Zap buttons to the first page
  (discussed with Nadya, 2/3). I'm not sold on this idea, mainly because space on 1st page is already very crowded, and likely to become moreso.
• *- option to automatically clear any prior output table contents on each Run
  This would usually be the most convenient behavior, but the initial implementation always simply appends. If we supported this option, there would be less need for an explicit Clear Output command.
• *- smarter detection of simple field renaming cases
  Should catch cases where attributes and sequence are unchanged, and old field name no longer exists.
• *- support buffering options, including table-level Save/Undo pushbutton(s)
  Especially relevant to use in append mode. Also may come in handy for supporting a way to preview the transformation. However, we may also want buffering disabled or reduced (from table to record-level), for the sake of performance. Remember to watch out for complications in use of form close box while buffered changes are pending.
• *- drag and drop from one grid/list to another for creating simple mappings
  We may want to introduce a listbox of input fields to make this easier if dragging between grids is not satisfactory.
• *- provision for ordering options on the source file, which determines output seq
  Support use of indexes. We could tie in generic MDA Ordering dialog here (from MDA110).
• *- support saving/restoration of grid column arrangements
  This might be handled best by creating a generic saver-based grid class. Otherwise the logic should go into the xformdbf-specific Saver subclass, SaverXformdbf.
• *- provide a way to see full input file spec. if longer than textbox
  This is an annoyance due to the inability of VFP's GETFILE() to display a default file name. While PUTFILE() can do this, it's not an appropriate substitute.
• *- maybe have a configuration option to toggle retention of deleted field mappings
  The current default behavior is to retain non-empty mappings for missing fields, unless they are explicitly deleted.
• *- optional deletion hiding toggle
  But it will do for starters to just SET DELETED ON by default.
• *- user-settable flag property to indicate whether EXACT should be set ON or OFF
  Wait and see if this is needed. Default is EXACT = OFF (scoped to data session).
• *- provide a pushbutton to modify the template table structure
  Nancy mentioned this on 1/14/00. It would be convenient, but not essential.
• *- support optional creation of output in a DBC
  Currently this can be accomplished only by opening an existing output table that has already been added to a DBC. This may be sufficient, but it would be nice to be able to create a new output table from scratch and explicitly assign it to an existing DBC.
• *- apparent VFP bug causing itemtips not to work for the mappings listbox
  Unfortunately, the itemtips feature doesn't seem to work, so long expressions are truncated and the only way to see the full expression is by double-clicking to bring up the VFP expression builder.
• *- support optional automatic sort order for mapping array
  As opposed to explicit sorting action, which is currently supported by clicking on one of the listbox column header pushbuttons.
• *- assign control source for meta-file SAY-only field AFTER the Saver is added
  To avoid flaky behavior of displaying the wrong file name. Wait and see if this problem reappears.
• *- quick report(s) summarizing DBF structure and mappings used
  Could be handled via generic Reporter facilities, with convenient tie-ins into the XFormDBF form.
• *- optional index creation steps on the output side
  This would be a sizeable complication. Decide if it really belongs in the XFormDBF facility, or should it be treated as a separate problem. I.e. we really could use a generic table-driven Indexer application. On the other hand, why not just let the user handle this via VFP's table designer. The basic assumption in xformdbf is that the template DBFs already exist, or you can create output file file VFP table designer.
• *- provide a way of returning the final functionid for the configuration used
  Allow for a mechanism to return this information, e.g. as an extension to the ErrHandler information already supported. The current design of Job Control no longer needs this feature (discussed with Nancy on 12/2/99), so hold off on its implementation until a clear need arises.

Optimization-related XFormDBF issues

• *- try using INSERT - SQL instead of APPEND + REPLACE commands
  This may be faster, and may have less severe limits maximum on command line complexity. May as well consider this an alternate optimization option, since we've already worked out logic that uses APPEND/REPLACE.
• *- see if there's any way to process the whole job in a single query
  I.e. try to eliminate the outer loop over input records. This may be doable in at least certain cases. Requires further analysis.
• *- see if use of SET RELATION is a faster way to get to lookup tables
  As opposed to use of indexseek() in mapping expressions. Requires some benchmarking.
• *- see if speed of non-interactive run/runc is affected by active page of form
• *- further optimizations for non-interactive Run/RunC modes
  We've already made the optimization of eliminating state-saving for these cases. Additional optimizations could be made, since there is no need for grids and visual form elements, but it's not yet clear this is worth the trouble. Note however that the fact that run is faster on non-grid pages suggests that this may be worth investigating further: it could make as much as 15% difference in speed.
• *- limit individual mapping lengths to some max allowed value, if there is one
  Whatever the restriction, this should be properly enforced to avoid unexpected errors. Note that the first pass at new xformdbf.XFOM_BuildRepl method will impose a limit somewhat less than the max macro length limit, to allow for the ",fieldname WITH " part of a REPLACE clause. Since macro length limit no longer seems to be a problem in SP3, hold off on this until clearly an issue.
• *- reduce the number of &-macro continuation lines for non-compiled mode
  Since we should be able to greatly increase the &-macro length limit, according to latest tests under SP3. Not important, because this old logic is not normally enabled, and may ultimately be discarded.
• *- support long-term retention of compiled transformation programs
  Avoids the need to recompile. This may not be worth the effort. Focus on other optimizations first.

Related issues pertaining to other applications:

• support generic error-handling in Saver and its subclasses
  This raises the issue of complications in dealing with 1-time errors associated with conversion of old configurations.
• generic Saver-based options for auto-save and prompt whether to save config
• eliminate direct dependence on other subdirectories like deduper and alansamp
  Example should be adjusted as necessary to keep demos small and self-contained, except for use of standard BT lookup tables, whose DBC is assumed to be on the default path.
• maybe extend commandr() to support macro-defined functions
  It's not clear that this really requires a Commander extension, or perhaps it's just a matter of usage. These would return something suitable for inclusion in a Rushmore-optimized query, avoiding the need for SQL to make calls to commandr(). Macro functions should return an expression without evaluating it, i.e. the result of a macro function is an expression. An SQL statement could thus be assembled from pieces (subexpressions) obtained via commandr(). But SQL isn't an issue for xformdbf, because it runs in a SCAN loop, not via SQL. Using macro-expanded functions should make it possible to perform repeated evaluations more quickly, so it provides a faster way to run DBF transformations.
• *- generic progress-indicator utilities
  E.g. uses status bar text if available, else WAIT NOWAIT window when status bar is turned off. Also could use push/pop message stacking features.

General guidelines and conventions:

• @- maintain clean copies of selected files in shared mdacomm directory
  Only copy the app, h, txt, and any other files that may be directly used by other apps. Don't copy files used only internally within the app.
• @- use functionid = 'xfbtwebpro' for BT/WebProperty layout sample configuration
• @- non-interactive modes should always avoid prompting
• @- typical usage generates output to a free table, but allow for output in DBC
• @- allow for arbitrary field xformations, including via commandr()
• @- state saving/restoration - deal with this as we go
  Because it will be helpful in testing, and it's a required feature.
• @- maintain consistent usage notes between wrapper function and corresponding form
  For this app, that means XFormDBF.prg and corresponding XFormDBF.Init.
• @- maintain thorough error and user escape-handling provisions
• @- avoid unnecessary direct cross-app dependencies by making use of mdacomm
• @- all apps should avoid potential name conflicts in case moved into mdacomm
  E.g. avoid file names like "notes.txt" in favor of "<appname>.txt".
• @- avoid using the same prefix for macro names and anything else (e.g. global mvars)
  Remember that VFP is case-insensitive with respect to names, so take care not to accidentally use a macro name unintentionally by sticking to a unique prefix.
• @- maintain comments in history of each app as to when last uploaded to BT server

Change history:

• 990720 btcrit should also support optional meta-file path flavor of arg3, cmdrob_arg
• 990813 revise Init method logic as to use of global default XFDB_CMDMV
  Really should use this only if the caller omits an explicit path to the Commander meta-file, even if XFDB_CMDMV is non-null.
• 990813 disable form resizing until auto-resize logic is implemented
• 990813 provide a quick init pushbutton for obvious field-to-field mappings
• 990814 generic mdacomm\getx.prg: treat all numeric field types as 'N'
  VFP's GETEXPR allows you to specify a data type or 'I', but no numeric expression will ever satisfy it for this choice, because type() always returns 'N'. Avoid this complication by handling it in the generic wrapper function getx().
• 990814 generic mdacomm\getx.prg: treat 'C' or 'M' field types as 'CM'
  I.e. allow for either character or memo expressions when either of these individual type codes is passed, because VFP's GETEXPR imposes a seldom-useful restriction, which is not consistent with its handling of numeric types. VFP's GETEXPR supports a multi-type form of specification, although the VFP Help documentation does not make this clear. E.g. specifying type as 'CM' will allow either Character or memo expressions, whereas 'C' or 'M' individually would not.
• 990814 display progress indicator (in status bar) during Run processing
• 990814 allow user escapes from Run processing and handle interruptions cleanly
• 990814 display summary timing statistics at end of each Run
• 990814 quick way to clear output file (but don't make accidents too easy)
  Explicit pushbutton to clear the output file would be handy, but an auto-clear option would make this less essential.
• 990814 new Test command to run transformation against a single input record
• 990816 suppress validation on Input initialization/cleanup expressions on Config page
  Validation produces spurious errors on usage of UDFs, and it causes problems with expressions involving "THISFORM", etc.
• 990816 don't include ..\commandr\commandr.vcx directly - use mdacomm copy
  Oops. Discovered this mistake while debugging.
• 990816 provide a way to clear initialization/cleanup expressions on Config page
  Current behavior treats empty response to getx() dialog as equivalent to Cancel, i.e. makes no change. Let a blank value of length > 0 be treated as a response, not as Cancel.
• 990816 illustrate Commander-based initialization command line
  For more manageable way of establishing database setups with lookup tables, especially if max expression length limit is encountered. Created sample database setup procedure under functionid = 'xfdbsbt01', using the same meta-file that xformdbf uses for state-saving. The connection to this Commander function was accomplished in the xformdbf configuration for creating a WebProperty layout by using the following InitExp: 
  thisform.saver.statecommander.DoFunc('xfdbsbt01')
• 990917 make adjustments for move from M: to D:
  This entails the following steps:
  - adjust icon for launching VFP to start in dir on D:... and use -cd:... option
  - config.fpw should be ok as-is - we already eliminated hard-wired drive/path
  - open .pjx and tell dialog OK to move it from M: to D:
  - confirm that all files in project have been relocated to D:, as expected
• 990917 sample xformdbf configuration, xformdbfdf, should avoid hard-wired M:
  Commander-based setup function, xfdbsbt01, also needs to be adjusted to avoid hard-wired drive/path. Take advantage of the new convention that LOOKUPS.DBC is always on the default path (for BT).
• 990917 suppress TALK in Load event
• 990917 new template output file specification on the Config page
• 990918 new command button to Clone template output table (if defined)
• 990918 prompt msg. for functionid should mention special empty value case
• 990919 support optional input/output file arguments
  These override any previous saved conguration's choice of files. Disable the Input File... and/or Output File... selection command buttons when there is an explicit argument override in effect, and also force any file overrides to stick even if another state is restored. The disablement of these command buttons will serve as a visual cue to the fact that caller-supplied argument overrides are in effect.
• 990919 new 'RunC' action, meaning "clone template and run non-interactively"
• 990920 suppress default prompt to replace for Clone in non-interactive mode
  Non-interactive modes should always avoid prompting.
• 990920 Run (and RunC) actions should bypass state-saving
  Which is faster and also avoids altering the template file specs or any other configuration parameters. Document this behavior.
• 990920 provide conspicuous summary documentation of available action codes
• 990920 hide dialog and display progress indicator during non-interactive Run/RunC
  This required an adjustment to the generic base SaverForm class, in order to eliminated restoration of form's Visible property, which is now handled in generic subclass SaverFormHideable (only used by the demo Saver application now).
• 990920 better progress indicator for both interactive and non-interactive cases
• 990920 clean out garbage test configurations and save latest xfbtwebpro config
  In the sample meta-file, xformdbf.dbf. Also pack to shrink overblown .FPT file.
• 991003 use more robust logic to suppress state-saving in non-interactive modes
  By performing the step, this.saver.statefuncid = '', in Init instead of Refresh method.
• 991003 fix logic to suppress file replacement prompt in non-interactive modes
• 991003 provide in-progress message coverage during form termination
  Just set status bar message during form Destroy, but keep residual WAIT WINDOW message giving summary of final Run statistics.
• 991003 break out the Run action as new method, xfom_run, which returns a success flag
• 991003 revise specs for return value from xformdbf to just return a success flag
  Previous implementation returned a count of records processed successfully. Screw that. If we need more than a success flag, use global mvars to return extended info. Wait and see what's needed.
• 991003 suppress pausing on errors for non-interactive mode, using new xfom_errmsg method
  Old logic used WAIT WINDOW without regard for interactive vs non-interactive modes. Adjusted to avoid pausing, and just return success flag result as .F.
• 991003 update the sample bldmstr.dbf structure in ...appl\xformdbf\
  Use the latest structure recently picked up from BT, and append sample data into this. Adjust mapping for output field OWNERZIP to map to new input field OWNERZIP5. Update saved sample xformdbf configuration under functionid = 'xfbtwebpro'.
• 991019 fixed bug in xformdbf.xfom_getout causing failure to display some fields
  Thanks to Nadya for both pointing out the problem and correctly suggesting its cause (the old SET EXACT trap). This bug caused some instances of fields with common prefixes to not get listed on the Fields tab, despite being present in the output file. Delivered this changed to BT server on 10/19 as a rush fix.
• 991019 rebuild with updated Saver stuff in mdacomm
• 991019 substitute new class-based Save/Restore buttons, and add new Config... button
  On bottom of form, i.e. as free-standing buttons visible regardless of current page.
• 991019 rename the old Config page to Options, and add new Config page for Saver stuff
  Introduce new member PageConf2 for Saver "Config" params, and change caption on old PageConfig to "Options"; keep old member names to minimize coding changes required.
• 991019 put all Config buttons on main form and hide them from new Config page
  Since they are redundant, and extra space will go to good use.
• 991019 remove redundant Saver-related info from the Options (formerly Config) page
  May as well increase some other field sizes while we're at it. Also must eliminate state saving logic for enablement of txtfuncid field, now eliminated.
• 991019 allow for brief title, category, and notes fields, which can be used for annotation
  (Alan's #3 priority, 9/27). Notes editbox could go on the config tab. Title could be part of the form window title (caption). These would be properties that are saved and restored, of course. This could be used to generate a query against the meta-file listing other available configurations, which could support a combobox for picking from canned configs (Alan's #2 priority, 9/27). Also could be used for supporting a summary report of available configurations (not yet done). 
• 991019 display configuration title in window title
• 991020 trim trailing blanks from title
• 991021 invoke new savrcm_setsaver method in form Init to handle post-Saver inits
• 991022 incorporate latest additional Saver refinements
  Most notably including the New Functionid textbox in the Switch To... dialog, and disablement of direct editing of functionid in other contexts.
• 991022 disable generic Saver-based form Save/Restore config when funcid is blank
  Also applies to btcrit.
• 991022 **** UPLOADED CHANGES TO BT SERVER'S D:\REDP\APPL\XFORMDBF\ & ...MDACOMM\ ****

• 991026 mention the advisability of frequent packing of Saver meta-files
  Because these tend to grow inordinately large .FPT files after many Save Config actions. Note that this is a generic Saver issue, which also applies to BTCrit.
• 991029 track down and fix cause of truncated title and category in savrpick.scx
  Flaky problem reported by Nadya on 10/26. I saw this once during testing, but it went away and didn't come back, so I assumed it was a transient one-time flakiness, possibly related to conversion of old configs. Evidently not, since Nadya's sample reproduces this problem consistently. This is really a generic Saver issue; additional notes are in Saver.txt.
• 991029 fix intermittent problems with missing mapping expressions for similarly named fields
  Nadya reported this problem and tracked it down to a SET EXACT issue; she supplied needed fix (substitute "==" for "=") in method xfom_makeout. Oops! Thanks again, Nadya.
• 991029 review/fix refresh logic bearing on command button enablements
  In testing on BT server, I noticed that the Test button is getting disabled and not re-enabled. Look into this and possible similar issues for other command buttons not being properly refreshed. Fixed this problem by refreshing the Test button enablement in the Input File grid's AfterRowColChange event. No other buttons are affected.
• 991029 invoke cleanupexp upon form termination
  Previous logic only invoked cleanupexp when closing and re-opening the Input file. There should also be a final invocation upon form termination, to allow for final cleanup, such as closing the DBC (which VFP fails to close despite having been opened in a private data session!).
• 991029 **** UPLOADED CHANGES TO BT SERVER'S D:\REDP\APPL\XFORMDBF\ & ...MDACOMM\ ****

• 991116 rebuild and test with new Saver features
  This includes the new Abandon, Copy, and Delete configuration buttons. Move Config... button from main form into Config page, to reduce needless clutter. Leave Abandon only on the Config page to reduce chance of accidental loss. I can move this to the main form body if users prefer; wait and see.
  991116 use the AGAIN option in openings of input and output files
  In case of secondary openings via initexp, as Nadya encountered in use of Towns table as input file. The USE statement's AGAIN option is virtually always what one wants, and does no harm.
• 991117 invoke prior config's cleanupexp prior to switching to a new config
  This only matters if the different configs have different cleanup expressions, which has not been the case in sample BT configs created thus far. It may not be an issue in the future for BT, because a common cleanup procedure may suffice indefinitely. But in the most general case, my intention was to allow for different cleanups, so this bug really should be fixed. Confirmed this fix under sample config named test1, which adds tracing to the cleanupexp.
• 991127 redefine form xformdbf.scx to be based on class ErrHForm
  Easily done by direct editing of the .SCX as a table: locate the record for objname = 'xformdbf'; replace field class (formerly = 'form') with 'errhform'; replace field classloc (formerly empty) with '..\mdacomm\errhandl.vcx'.
• 991127 initialize selected xformdbf.ehp_... properties for error reporting
  ehp_appname, ehp_targclass, ehp_targlib, ehp_targname (= 'xformerrh'), i.e. the external errhandler object will be allocated and assigned to global mvar m.xformerrh.
• 991127 eliminate obsolete properties xfop_noninteractive and xfop_result
  By substituting use of generic errhform properties ehp_interactive and ehp_success.
• 991127 initialize xfop_template to a blank string (not .f.) via property sheet
  This was a bug, which causes a state-saving error when you attempt to save a brand new configuration.
• 991128 eliminate obsolete method xfom_errmsg
  By substituting use of generic errhform.ehm_showapperr() method. This change affects the following methods: Destroy, Init, xfom_openinput, xfom_clone, XFOM_Run.
• 991128 use VFP's newobject() function and method where appropriate
  To avoid annoying hassles with older SET CLASSLIB + add/createobject() approach. This change pertains to logic in xformdbf.Init.
• 991128 xformdbf.init changes - prelim.:
  - compare with errhandl.init (demo form under ...\appl\errhandl\)
  - determine this.ehp_interactive and windowtype from action arg first
  And do this carefully, because err-handling is not yet initialized.
  - invoke dodefault() to handle standard err-handling inits
  - use this.ehm_showapperr() to report invalid Action errors
  And likewise for any other explicitly detected errors.
  - use generic this.ehp_interactive property where appropriate
  - avoid SET CLASSLIB by using newobject() method
• 991129 fixed bug causing failure to properly enable/disable Zap button
  Added thisform.refresh to the Clone button's click method - oops.
• 991129 more xformdbf.init changes:
  - debug usage of newobject(): requires additional arg before class Init args
  - be sure to check for errors after any calls to error-detecting methods
  - beef up abnormal error trapping and clean reporting
  - clear functionid to suppress state-saving for error abort cases
• 991129 xformdbf.refresh: invoke dodefault() to pick up generic errhform class logic
• 991129 output table is not getting closed upon form termination
  It persists in a residual private data session after the form terminates. Fixed by adding explicit closing logic in xformdbf.Destroy method. The residual data session turns out to be a subtle consequence of allowing xformdbf to allocate an external errhandler object that persists beyong the life of the form. If the caller does this allocation instead and assigns this object reference to m.xformerrh (the default ehp_targname), then xformdbf will use the caller-supplied errhandler instead of allocating one, and there is no residual data session.
• 991129 xfom_openoutput: add error-trapping around the USE statement
• 991129 xformdbf.Unload: let errhform class default prevail
• 991201 xformdbf.XFOM_Run:
  - thisform.ehm_zero() before running
  - use thisform.EHP_TrapErrors = .t. surrounding the error-prone logic
  - test out all recognized error cases
  - note that this also takes care of any invalid mappings
  Which may have gotten past the validation performed when mappings were originally entered, e.g. due to subsequent changes. No additional validation step is needed, because these errors are now trapped cleanly in Run logic.
• 991202 support error-handling for the Zap command button
  These requires a new form-level method, xfom_zap, for proper error trapping.
• 991202 break out core method, xfom_run1, to perform transformation on a single record
  This logic was previous done directly within the main xfom_run method, consisting mainly of an inner loop over fields. By breaking it out as a separate method, we can avoid redundant code otherwise required to support both the Test and Run command buttons.
• 991202 support error-handling for the Test button
  This requires a form-level method for proper error trapping. While we're at it we really should create a core method, xfom_run1, which should be usable by both the Test button and xfom_run.
• 991202 check for abnormal failure cases of xfom_openinput and other error-checking methods
  These errors should be preserved if they lead to overall failure. E.g., we don't want prior error to be superseded by "Input file must be open", which is really the result of a preceding error.
• 991202 provide error-handling for the Create/Modify structure command button
  Introduce new form level method, xfom_modistru, to do this with proper error trapping.
• 991202 fix/improve error handling, esp. for file opening, initialization, and Run errors
  Support clear, consistent conventions for error return values. Provide a clean way of returning extended error information, e.g. via global mvar(s) to return error message, etc. Use xformdbf.h for corresponding error code/message macros. Use the new generic MDA error handling classes (introduced for Querier and extended for forms) to deal with this cleanly. Completed a full pass over all basic error detection throughout xformdbf, except for generic Saver-level error-handling.
• 991203 add logic for null-checking (see conv. w Nancy on 11/17)
  Introduce a new flag property for this option, xfop_nullisblank, and set this to .T. by default. Adjust xfom_run1 logic to treat null output field values as if they were empty when xfop_nullisblank = .T. Eventually we might want to allow the user to toggle this option on a per-config basis, but this is not an immediate requirement for BT.
• 991203 adjust wrapper function, xformdbf.prg, as nec. for clean error handling
  Compare with errhandlr.prg. Support correct success flag result for modeless cases. Avoid encountering errors in the wrapper function, so that all validation and error-handling can be localized in the form logic.
• 991203 beef up argument validation in xformdbf.init
• 991203 **** UPLOADED CHANGES TO BT SERVER'S D:\REDP\APPL\XFORMDBF\ & ...MDACOMM\ ****

• 000105 optimize performance by eliminating field-by-field replacement
  Make this mode optional, controlled by property xfop_replfieldbyfield, because it doesn't provide as much useful diagnostic error info as to the specific invalid mappings that caused failure. Also note that there exist certain limitations on the use of optimized logic due to various VFP (undocumented, of course) limits, e.g. the max number of fields involved in a single replace statement.
• 000105 let the Test button always use the unoptimized mode
  Since this provides better field-level validation, and performance is not an issue.
• 000106 use shorter aliases for primary input and output tables
  This may help to alleviate problems caused by VFP size limits. Changed macro XFDB_ALINP from 'xforminp' to 'x', and XFDB_ALOUT from 'xformout' to 'y' (in xformdbf.h).
• 000107 user selectable optimization checkbox, chkNoOptim, on the Options page
  Toggle for form property, xfop_replfieldbyfield. Also support related state-saving logic for this new control.
• 000107 run more benchmarks and note stats in comments attached to various configs
  Compare speed of bound and unbound app in both field-by-field and optimized modes. Speedup varies, but a factor of 5 improvement seems to be the typical effect of using optimized mode.
• 000111 test to confirm exact &-macro length limit
  VFP Help seems to imply it's 8192 chars, but my recollection is that it's more like 254 chars. Maybe this limit has been increased since I last checked, so test it again. This limit really should be defined as a macro in commandr.h. Preliminary testing indicates that the &-macro limit has been relaxed in VFP6 SP3, so it may not be a problem any more (other limits get in the way first). Hold off on further testing of the &-macro limit until the problem comes up in a clear case.
• 000111 support optional generation and compiling of core transformation procedures
  This is now permissible in VFP 6 SP3, thanks to runtime support of the COMPILE command. Use the new mdacomm\cmdprg.prg utility to handle this. Benchmark this approach against previous optimized mode, which used &-macro expansion.
• 000111 introduce form properties for VFP limits that require experimentation
  This will make it much easier to determine the proper default settings empirically, since these are tricky to test and may change from VFP version to version.
• 000111 adjust optimized logic to circumvent the VFP limit on max # of REPLACE fields
  This could be done by breaking it up into several successive REPLACE commands.
• 000111 sort out cmdprg() error cases properly
  Really should eventually be done by introducing new cmdprg.h for codes and messages, but preliminary hard-coding will suffice.
• 000113 determine the exact limit on max # of REPLACE fields
  Preliminary testing suggests that this limit is between 100 and 200 <field> WITH <value> clauses, based on configs test6, test9, and test10. Further tests indicate the exact limit is 128 clauses. Test indicates that use of more complex mappings does not cause a problem, i.e. VFP's limit of 128 on the maximum # fields that can be handled in a single REPLACE statement is not dependent on the complexity of the mapping expressions. I also tested to see if VFP's setting of SYS(3055), FOR and WHERE Clause Complexity, has any effect on the max # fields limit: it doesn't.
• 000113 adjust progress indicators to be more efficient and informative
  Nadya requested more info. in email of 12/28. Note greater need for information when running in non-interactive mode, since screen is not visible. Revised logic to use a static WAIT WINDOW message + odometerized status bar message. This made a noticeable performance improvement.
• 000113 review recent email from Nadya & Nancy for any additional points to be noted
• 000113 discuss eliminating special NULL-handling logic for optimized modes
  Currently we support automatic treatment of NULLs as equivalent to empty values, to avoid errors in output fields that don't support NULLs. We could easily make this a user selectable feature on the Options page, but the current implementation only allows this option to be set via xformdbf's Property Sheet. For the optimized mode, which eliminates the need for field-by-field processing, such a feature would require a fair amount of extra programming effort and yield significantly slower performance, so I'm inclined to avoid this approach. Instead I would advocate either eliminating the use of NULLs altogether, or using suitable mapping expressions to convert NULLs to blanks if the output field doesn't allow NULLs. Alan suggested something along the lines of the denull() utility.
• 000114 new generic mdacomm\denull.prg utility for eliminating NULL values
  By using this to pre-process input files, we can eliminate the need for special handling of NULL input values by xformdbf, which would be a royal pain to optimize.
• 000114 disable the xfop_nullisblank logic in xformdbf, by default
  This should no longer be needed if the caller uses denull() to pre-process input files (when necessary). Invocation of denull() could be handled as an extra job step to follow outer join queries, which introduce unwanted NULL values.
• 000114 **** UPLOADED CHANGES TO BT SERVER'S D:\REDP\APPL\XFORMDBF\ & ...MDACOMM\ ****

• 000303 fix mdacomm\getx.prg to perform appropriate type-checking for all field types
  This relates to the problem Nadya encountered in xformdbf, where an attempt to use a mapping of the form ctod(<character field>) was disallowed, even though the output field was a date field, giving an error message of the form "Expression must be of type 'N'". The bug lay in getx.prg, which upon further review and testing, I realized was flawed in a more general way. To fix these bugs, I am revising and clarifying the specifications of getx's handling of its 2nd argument, the type_arg, so as to be suitable for any standard VFP field type. When a standard VFP field type is specified, getx expands this type code into a multi-character GETEXPR type code that allows the user to supply any type expression that a field of the given type field can be REPLACEd with. E.g. a character field can accommodate either 'C' or 'M' expressions, any numeric or currency flavor of field can be replaced with another flavor of numeric or currency type, etc.
• 000303 SET CENTURY ON in form Load event
  (discussed w Nadya on 2/18). Also take this opportunity to allow for propagation of Load event to parent class, in case of future logic in class ErrhForm.Load.
• 000303 Abandon button should be on the main form body
  (discussed with Nadya, 1/26) Also hide Abandon on the Config page, since it's redundant now.
• 000303 add support for hot keys
  (discussed with Nadya, 2/3). I made a full pass over the xformdbf form, including page tabs and page bodies, except for the Config page, which should be done in Saver first.
• 000303 **** UPLOADED CHANGES TO BT SERVER'S D:\REDP\APPL\XFORMDBF\ & ...MDACOMM\ ****

• 000316 introduce editbox for viewing/editing mapping expression directly
  (Previously we only supported entry of mapping expressions via VFP's expression builder dialog.) Make use of generic mda splitter here (see ...\appl\splitcon\). Compare with ...\appl\saver\savrpick.scx, a real example of an application of this splitter mechanism. Start by adding an instance of the Splitcon_2_1 subclass onto the Fields page, and move lstOutfields into the upper subcontainer, Splitcon_2_1.subcon_1_1. Adjust all old refs to lstOutfields for moving this member into ...subcon_1_1. Add resizing logic to each subcontainer, plus force initial resizing of splitcon in the form Init method. Use the listbox's When event to force proper refresh of the editbox.
• 000317 state saving/restoration of splitter position on the Fields page
• 000317 preliminary validation logic for edMapping
  Break this out into a new form-level method, xfom_validmapping, for proper error handling that can cope with UDF cases. Note that this method may eventally be useful in other contexts, but for now it's only called by edMapping.valid.
• 000317 disable treatment of Escape as equivalent to clicking on Exit
  Because this could have unintended consequences. Allow Escape to have the more useful behavior of simply reverting edits to the current field, e.g. the mapping editbox.
• 000318 further refinements to new editbox for mapping expressions
  - refresh listbox if edMapping validation succeeds
  And test that other refresh cases are covered, including changes made via getx() by double-clicking on a row of the listbox.
  - delete mapping & quick mapping buttons should force an edMapping refresh
  - spurious validation error on empty mappings - oops - no-op for these
  - check that hot keys don't circumvent or conflict with validation
  - type validation has stopped working properly
  Oops - fix typo in recent edMapping.valid logic changes.
  - mapping type validation errors should have suitable err msgs
  - VFP error message for bad mapping should include the expression evaluated
  - add logic to suppress "Var not found" in mapping validation if input is closed
  Don't assume input file is open. If not, we should loosen validation to disregard "Variable not found" errors (VFP Error # 12).
  - test out behavior on cases with placeholder mappings
  I.e. where output field doesn't currently exist, but mapping was retained. In such cases we have no field type to validate against, but we should still check that the result is well-defined.

Copyright © 2000- 2002, SpaceTime Systems