ASAP Troubleshooting

Where to look when addresses do not reach the ASAP Address Verification Queue (program #1675), do not get verified, or an incident cannot be sent to a PSAP. Every configuration item referred to here is described on the ASAP Setup page.

Statuses in the queue

The check boxes in the top right of program #1675 filter the queue by status. The queue opens with Failed selected, so an address in any other status does not appear until you select that status and choose Search.

Addresses that are not shown under any status

One condition holds a record in the queue tables without displaying it under any of the four check boxes:

• The agency's submission record. Each agency gets one record when you answer Yes to the submission prompt. It is consumed once the Dispatch Service expands it into individual installation entries. If that expansion produced no addresses at all, the record is listed under Failed with an empty Installation Number, so you can see that the agency was submitted and nothing qualified. Select it and choose Reset to submit the agency again once the setup below is corrected.

An address whose send to the broker did not complete is not hidden. On APIEngine 08.92.0024, 08.95.0009 and later it is moved to Failed and the reason is in the APIEngine log; on earlier builds it stays under Verifying. See Addresses stay in Verifying below for how to tell a send that never left from one that is waiting for an answer.

The submission record can be cleared from the screen. Select the row and choose Reset, which returns it to Ready to Process so the Dispatch Service picks it up on its next pass. Resetting the agency's submission record resubmits every address for that agency, so correct the setup below first — otherwise the same addresses are filtered out again.

If an installation you expect is in neither the queue nor any status filter after that, contact Innovative Support with the agency and installation numbers.

No addresses appear after onboarding an agency

Work through these in order. Each one stops submission silently.

1. Select the Ready to Process check box and choose Search. Program #1675 opens with Failed selected, and newly submitted addresses arrive in Ready to Process, so they do not appear under the default filter. This is the most common cause.
2. Confirm option asap is on and its stored procedures have been recompiled. A static option that has not been recompiled behaves as though it were off, while the option screen shows it on.
3. Confirm you are on the primary database. On a backup host nothing is submitted, sent, or received.
4. Confirm the agency record is correct in program #1546, #1547, or #1552: Address Type is ASAP in uppercase, and Address 3 holds the nine-digit code, exactly nine characters.
5. Confirm the agency is a police, fire, or medical agency. Other agency types carry the same fields but do not participate in ASAP.
6. Confirm you received the submission prompt and answered Yes. The prompt appears only when Address Type changes to ASAP during that save. If it never appeared, clear Address Type, save, set it back to ASAP, and save again.
7. Confirm the agency has not already been submitted. Each agency is submitted once. A second Address Type change produces no prompt and no new entries. If the agency was submitted during earlier testing, its accounts were already evaluated at that time — and any account that was not eligible then was skipped and will not be reconsidered by repeating the save. Make the accounts eligible using the items below, then change each account's monitoring status to bring it into the queue, or contact Innovative Support to have the agency's submission record reset.
8. Confirm the Dispatch Service is running in the SBN Service Manager. It performs the expansion and the sending; no other background task is involved.
9. Wait one interval. Addresses arrive in batches sized by the agency's Addr ID (1) and Addr ID (2) fields, so a large agency fills the queue gradually.

Some installations appear and others do not

The missing installations are failing one of the per-account requirements. All three are checked silently.

• The agency is not on the installation's action plan. Open the installation in Data Entry (program #559) and confirm the agency is on the action plan as the police, fire, or medical dispatch agency.
• The monitoring status does not have Include in PSAP selected. Open Monitoring Status (program #1540), find the monitoring status the installation uses, and confirm the Include in PSAP check box is selected. Installations on a status without it are skipped.
• The installation has no address. An account with no address on file cannot be submitted.

After correcting any of these, change the installation's monitoring status to bring the account into the queue.

Addresses stay in Ready to Process

The addresses are queued but are not being sent.

1. Confirm the Dispatch Service is running in the SBN Service Manager.
2. Confirm the Dispatch Service is pointed at the correct APIEngine and that its credentials are valid. Its log records the address it calls and any failure to reach it.
3. Confirm APIEngine is running and reachable.
4. Confirm the agency's Addr ID (1) and Addr ID (2) are not set to values that release addresses more slowly than you expect. Blank fields mean 5 addresses every 60 minutes.
5. Confirm you are on the primary database.

Addresses stay in Verifying

An address moves to Verifying when the Dispatch Service hands it to APIEngine, before APIEngine sends it. On APIEngine 08.92.0024, 08.95.0009 and later a send that fails moves the address to Failed and writes the reason to the APIEngine log, so a Verifying address there has reached the broker and is waiting for an answer. On earlier builds a Verifying address is in one of two situations, and they look identical in program #1675:

• The send never left APIEngine. The broker has no record of the address.
• The send reached the broker and no answer has come back yet.

The Dispatch Service log cannot tell them apart. It only shows the Dispatch Service asking APIEngine to send, and APIEngine answering that there are no addresses to send once the batch has already been handed over. Use the APIEngine log.

Confirm the address left APIEngine

APIEngine is the machine that talks to the broker. Outbound traffic to the broker leaves from the APIEngine host, not from the host running the SBN Service Manager, so firewall rules for the broker address apply to the APIEngine host.

1. Open the APIEngine log for the date shown in the Change Date column of the Verifying address. The send happens at the moment the address enters Verifying, so that date and time is where the record of the send is.
2. Look for a line beginning REQUEST to followed by the broker address, and shortly after it a line beginning RESPONSE from the same address with Status: 200. That pair means the broker accepted the message.
3. A line tagged Err: instead means the address never left. The wording says why: NETWORK error sending ASAP message or TIMEOUT sending ASAP message is a connectivity or firewall problem on the APIEngine host; RESPONSE from with a status other than 200 is the broker refusing the message, usually a certificate or identifier problem; ASAP XML failed XSD validation is a data problem with that address; Send XML exception with no REQUEST line at all means the send was abandoned before the network, which points at the ASAP block of the APIEngine settings - a blank or missing broker address, or a certificate path or password that cannot be loaded.
4. Check the broker address on the REQUEST line against the address ASAP gave you for this environment. Test and production brokers have different addresses, and a message accepted by the wrong one never reaches the agency.
5. Confirm from the installation side: once the broker accepts an address, the installation's Alarm Log receives an entry reading Address submitted for validation to the agency. No such entry means the broker never accepted it.

Two APIEngine calls confirm the current configuration without waiting for the next batch:

• The ASAP ping endpoint sends a test message to the configured broker address and reports success or failure. On APIEngine 08.92.0024, 08.95.0009 and later the reply carries the reason in plain language (the broker address that is missing or wrong, the certificate that could not be loaded, the host that could not be reached, or the broker's refusal) and the same text is written to the APIEngine log; the Dispatch Service log shows the same reason because it prints APIEngine's reply.
• The APIEngine diagnostic endpoint checks that the certificate file in the ASAP settings exists, is readable by the APIEngine account, and opens with the configured password.

Both are described on the APIEngine ASAP page.

After correcting the configuration, select the Verifying addresses in program #1675 and choose Reset. They return to Ready to Process and the Dispatch Service sends them on its next pass, at the rate set by the agency's Addr ID (1) and Addr ID (2) fields.

The address reached the broker and no answer has come back

The answer arrives as an inbound message from the broker, so this is the inbound path.

• Confirm APIEngine is publicly reachable over HTTPS from outside your network. The broker sends inbound messages directly to it.
• Confirm the certificate issued during onboarding is installed and has not expired.
• Confirm the broker holds valid credentials for your APIEngine, so its inbound messages are accepted rather than rejected.
• Confirm the inbound address you registered with ASAP resolves to this APIEngine.
• Check the APIEngine log for inbound messages arriving and being rejected.

An address is Failed

The agency reviewed the address and rejected it. Nothing is wrong with your configuration.

1. In program #1675, select the Failed check box and choose Search.
2. Select the installation and choose ASAP Validate.
3. Correct the address in the fields at the bottom of the ASAP Address Verification Detail window and choose Save. SBN resubmits it.

If the agency rejects a corrected address repeatedly, the address as the agency holds it differs from yours; take it up with the agency directly.

The eDispatch prompt does not appear in Dispatch

When an operator dispatches police, fire, or medical on an incident and no eDispatch prompt appears, one of the following is not satisfied:

• Option asap is off, or its stored procedures have not been recompiled.
• The incident's zone type has no value in the eDispatch field in Zone Types (program #1733).
• The installation's address has not reached Verified status for that agency in program #1675. An address in Ready to Process, Verifying, or Failed is not eligible.
• The agency for that dispatch type is not on the incident's action plan, or does not have Address Type ASAP with a nine-digit code.
• The incident has already been sent to that agency.

The eDispatch Chat button does nothing

If selecting Chat produces a message that the event option must be set, option asapevt has no value. Set it in Options (program #1887).

Nothing moves at all

When neither addresses nor incidents move in either direction:

1. Confirm you are on the primary database. This stops all ASAP processing silently.
2. Confirm option asap is on and its stored procedures have been recompiled.
3. Confirm the Dispatch Service is running.
4. Confirm APIEngine is running, reachable, and configured with the broker address and certificate.

What to send Innovative Support

If the queue is still not behaving after the checks above, include:

• The agency number and the program you configured it in.
• One installation number you expect to see in the queue and do not.
• Whether the submission prompt appeared when you saved the agency, and what you answered.
• Which status check boxes you searched in program #1675.
• Whether the Dispatch Service is running, and the relevant portion of its log.
• For addresses stuck in Verifying: the APIEngine log covering the Change Date of one of them, and the result of the ASAP ping endpoint.