Skip to main content

Reveal Processing

Reveal AI Integration API

(NIA for short)

September 9, 2021

1 New MSA Instructions

Begin by editing the NIA.config file. It will be on the DB server at a location like D:/Reveal/NIA/NIA.config. You should see several sections like this:

[MSA - 000000]

User: <some username>

Password: <some password>

URL: https://trunk.revealdata.com/rest/api/

NIA needs a user and password configured to be able to log into Reveal, so copy/paste that config section and edit the MSA number, username, password, and Reveal API URL appropriately.

Generally the easiest thing to do here is to re-use the processing-user login we create for processing.

The MSA number should be the same number as set in the instance’s System Settings under the setting MSANumber.

You should not need to restart any of the NIA services for the updates to take effect. It re-reads the config before it uses the values.

On the Reveal side, the following system settings need to be edited to link into Reveal correctly:

  • NexLP API TenantName NexLP Tenant name for creating databases and interacting with Reveal AI.

    e.g. Default

  • NexLP API Database Suffix Suffix to place after new database names cre­ated within Reveal AI. Mainly important for all-in-one database server setups where we do not want review’s database name to conflict with Reveal AI’s.

    e.g. RevealAI

  • NexLP API Database Server Hostname for Reveal AI’s database. This must also match the name of a Reveal AI datasource in the Tenant you have set below.

    e.g. REVEALAI-SANDBOX-1

  • NexLP API Login User name with which to log into Reveal AI’s API.

    e.g. nexlp-admin@revealdata.com

  • NexLP API Password Password for the NexLP API user.

    e.g. fooBarBangBiz

  • NexLP API URL URL to Reveal AI’s API.

    e.g. https://nexlp-sandbox.revealdata.com/StoryEngineWebApi/api

  • NexLP Integration API URL URL to NIA’s API.

    e.g. http://10.0.130.56:5566/nia

  • MSANumber MSA number for the instance as previously mentioned.

    e.g. 000000

  • Reveal AI Version Reveal AI version number that is currently running.

    e.g. 2.80.

2 Network Requirements

Ports with a non-standard designation (5566) can be altered in code as necessary.

Source

Dest

Port

Required

NexLP Integration Worker

Reveal API

NexLP API

443

443

X

X

Reveal Index Batch

NexLP Integration API

NexLP API

5566

443

X

X

Reveal Ops Center

NexLP API

443

X

Reveal Web

NexLP API

443

X

NexLP Integration Worker

NexLP Database

1433

X

NexLP Integration API

NexLP Database

1433

X

NexLP Integration Worker

Email Server

465

Reveal AI integration will send out email alerts for high-level unhandled errors with syncing. This can be disabled if desired.

3 Troubleshooting Help

The following is a breakdown of the statuses of a NIA job:

processing_job_status_to_desc = {

-1: ’Unknown’, 0: ’Created’,

  1. ’Submitted’,

  2. ’InProcess’,

  3. ’Complete’,

  4. ’Error’,

  5. ’Cancelled’,

  6. ’CancelPending’,

  7. ’Deleted’,

  8. ’Modified’,

  9. ’Processing Job Submitted’,

  10. ’Deletion Job Submitted’,

  11. ’Deletion Identify Job Submitted’,

  12. ’Deletion Job Started’ }

You can check the current status of a particular job by running the following on the nia-database database:

select *

from nia_job_queue

select *

from nia_job

order by 1 desc

The first query gets you anything sitting waiting in the queue. Realistically, this should almost always be empty. The second query gets you a historic record of jobs which have status in the column nia job status.

Generally, the NIA services should be on the Reveal AI Database server located under:

D:\Reveal\NIA

You should also find a config file in that folder that you can double-check set­tings on. In that config file, passwords must have the dollar sign and percentage sign escaped by putting the offending character in twice.

To double check that the NIA API is running, you can navigate to the following:

http://10.0.73.128:5566/nia/version

Replace the IP address with the IP of the database server, or just localhost if you’re on the server itself.

4 Network Communication Flow

Definitions:

  • NIA API Runs on the Reveal AI DB Server under the name NexLP Integration API service. Our homegrown API.

  • NIA API Worker Runs on the Reveal AI DB Server under the name NexLP Integration API Worker service. A worker separate from our home­grown API but developed by us.

  • NexLP API Runs on the Reveal AI Web Server. Written by Reveal AI.

  • Reveal API Runs on the Reveal Web Server. Written by Reveal.

  • Index Batch Service Runs on the Reveal Lightweight Services Server.

4.1 Text Syncing

Initiated by a user indexing documents or deleting documents from Reveal.

  1. Index Batch Service sends a request to the NIA API via port 5566 to create a job. From this point forward, the Index Batch Service will continue to periodically poll the NIA API via port 5566 to get status on the job.

  2. NIA API Worker watches the nia-database for new work. When it sees the new job, the NIA API Worker queries the Reveal API via port 443 to get text metadata for the job.

  3. The NIA API Worker writes out the text it receives to a configured text storage location. At the same time it builds a load table on the Reveal AI Database Server inside the Reveal AI Source Database associated with the job.

  4. After the NIA API Worker finishes loading to the load table, the NIA API Worker queries the NexLP API via port 443 to create a processing job for the database.

  5. The NIA API Worker will continue to poll the NexLP API via port 443 until it sees the processing job is done.

  6. When the job is done, the NIA API Worker updates the status in the nia-database. This is eventually also returned to the Index Batch Service which finishes out text syncing workflow.

4.2 Score Syncing

Initiated periodically by the Index Batch Service.

  1. The Index Batch Service queries the NIA API via port 5566 to get a list of the newest model update jobs from a given starting time.

  2. The Index Batch Service checks these values against the relevant Reveal database to determine if it needs to get new scores from Reveal AI.

  3. For every case for which it needs to get scores, the Index Batch Service queries the NIA API via port 5566 to get the newest scores for a model.

  4. The NIA API queries the NexLP API via port 443 to get the model identifiers and the SQL database name for the Reveal AI Storybook.

  5. For each model, the NIA API queries the Reveal AI Storybook Database directly via port 1433 to get current scores for the model.

  6. The NIA API takes these scores and compares them to the cached scores in a configured location.

  7. The NIA API returns the new scores to the Index Batch Service, and also caches the scores it just read to a configured location.

  8. The Index Batch Service saves these scores into the Reveal database via port 1433.

  9. The Index Batch Service commits the changes to the NIA API by running a query on port 5566.

5 New Instance Instructions/On-Prem Installation

5.1 Storage Requirements

Reveal AI integration needs some sort of ephemeral network storage. When a new document sync job is submitted, the Reveal AI integration worker will start pulling text for a case locally to be loaded to Reveal AI. This means that the Reveal AI integration worker will need storage with at least as much space as the text size of your largest case.

The Reveal AI processing servers will ingest this text during processing. This means that the datasource must be visible by both NIA and Reveal AI’s processing services.

5.2 Setup Instructions

  1. Download the packaged services from here:

    s3://revealdata-software-private-build-repository/RevealIntegrationAPI/ NexLPIntegrationAPI.zip

  2. Unzip that and it should look like the following:

    Installation Logs

    NIA.config

    NiaAPIService

    NiaWorkerService

    Score Syncing

    Text Syncing

  3. Place the unzipped contents wherever you want the install to be. In this document we’re going to assume everything is located here:

    D:\NexLP\Reveal\NexLP Integration API\

    This root directory is important to note because it is used in further installation commands. We will call out when this is important.

  4. The Installation directory has two commands to use for installation and a C++ install redistributable:

    Install Commands.txt

    VC_redist.x64.exe

    This redistributable installer may not be necessary to install depending on what is already installed on the server. Safer to install though as it won’t hurt anything to have it.

  5. The installation instructions should look something like this:

    cat Installation/Install\ Commands.txt

    New-Service -Name "Reveal-NexLP Integration" -DisplayName "Reveal-NexLP Integration" -Description "Reveal-NexLP Integration" - BinaryPathName ’D:\NexLP\Reveal\NexLP Integration API\ NiaWorkerService\NiaWorkerService.exe -c "D:\NexLP\Reveal\NexLP Integration API\NIA.config"’

    New-Service -Name "Reveal-NexLP Integration API" -DisplayName " Reveal-NexLP Integration API" -Description "Reveal-NexLP Integration API" -BinaryPathName ’D:\NexLP\Reveal\NexLP Integration API\NiaAPIService\NiaAPIService.exe -c "D:\NexLP\ Reveal\NexLP Integration API\NIA.config"’

    Note

    Note the double-quotes around the configuration file path. This is neces­sary so be careful if you’re dealing with filepaths that have spaces in them.

    Note

    Note the four locations in the above where the root path appears.

    D:\NexLP\Reveal\NexLP Integration API\

    Make sure to change this root path if you have the files downloaded some­where else.

  6. Run the above install commands from an admin powershell to install the services. If you find you didn’t set the paths correctly you can run the following to remove the services for re-installing:

    Remove-Service -Name "Reveal-NexLP Integration"

    Remove-Service -Name "Reveal-NexLP Integration API"

  7. Edit the config file. In our example this would be at:

    D:\NexLP\Reveal\NexLP Integration API\NIA.config

    It should look like this:

    # Dollars signs need to be doubled to escape.

    [MSA - 999999]

    User: username

    Password: password

    URL: https://foo.hostname.com/rest/api/

    [NIA]

    Temporary Text Output: D:\NexLP\Reveal\NexLP Integration API\Text

    Syncing

    Worker Load Threads: 2

    Worker Batch Size: 2000

    API Log Location: D:\NexLP\Reveal\NexLP Integration API\Logs\nia_api.log

    Worker Log: D:\NexLP\Reveal\NexLP Integration API\Logs\nia_worker.log

    Verify SSL: False

    mass edit batch size: 500

    [NIA Score]

    Score Storage Path: D:\NexLP\Reveal\NexLP Integration API\Score Syncing

    [Database]

    Type: MSSQL

    [Review SSO]

    User: foo

    Pass: bar

    [MSSQL]

    NIA Database Name: nia-database

    Host: nexlp2

    Driver: SQL Server Native Client 11.0

    [NexLP]

    User: nexlp_username

    Pass: nexlp_password

    URL: https://nexlp.revealdata.com/StoryEngineWebApi/api/

    [email]

    enabled: False

    emailFrom:

    emailTo: to

    emailHost: host

    emailPort: 465

    emailPassword: Password

    emailUser: User

    From here the following sections need to be filled with the correct linking information:

    • MSA - 999999 The 999999 should be replaced with whatever the MSANumber system setting is set to in review. Make sure this setting is set in Review. The User, Password, and URL settings should be altered to point to whatever site you’re linking into.

    • NIA Generally these can be left as is as long as you edit the root paths. Again in this example the root path is D:\NexLP\Reveal\NexLP Integration API\. One thing to note is that wherever you point the Temporary Text Output setting needs to have enough disk space to sync down text for at least your largest dataset to sync.

    • NIA Score Edit the score syncing root path.

    • Database Nothing to edit here.

    • MSSQL The Host value needs to match whatever the Database server is named as within Reveal AI’s datasources. If you need to double-check, Reveal AI stores this information in the [nexlp].[DatabaseServer] table in the Name column.

    For the Driver value, as long as you’re installing this on the Reveal AI database server you can likely leave this as it. However, to double-check you can confirm the configured value is in the output of the following powershell command:

    Get-ODBCDriver

    That will print out the ODBC drivers installed on the server for use. Windows should, however, come pre-packaged with the configured value.

    • NexLP As with the MSA section, this just needs to be re-configured with the appropriate information required to connect to Reveal AI’s API.

    • email Edit settings here if desired. They are all self-evident. Email can be turned off by leaving the enabled setting at False.

  8. On the Reveal AI database server you’re running this on, create an empty database named nia-database. The API will automatically create the necessary tables in this database on startup.

  9. Start the API service followed by the Integration service. Do not run the Integration service if the API fails to start.

6 Build Instructions

If we need to re-build, put code on disk on a Windows machine somewhere. (e.g. D:/NIA/). Open up a powershell window. Run:

$env:PYTHONPATH = ’D:\NIA’

That will set your pythonpath. Then do:

cd D:\NIA\NIA\Service

pyinstaller.exe .\NiaWorkerService.py --clean --noconfirm

pyinstaller.exe .\NiaAPIService.py --clean –noconfirm

That should build the services into the dist directory. If it’s not already there, put a NIA.config file at the top of the dist directory.

7 Notes

If you get errors when trying to run the source because of some error like:

ImportError: cannot import name 'cached_property' from 'werkzeug' (C:\Program Files\Python37\lib\site-packages\werkzeug\__init__.py)

Run this:

pip install werkzeug==0.16.1

In this section: