Mohsin Kalam’s Blog

October 9, 2007

PAM

Filed under: R2 EDI — Mohsin Kalam @ 2:00 pm

Goal:

To describe various settings used by the BizTalk EDI runtime components stored in PAM.

Overview:

Partner Agreement Manager (PAM) is the central repository for the configuration settings accessed by the EDI runtime components during document processing. It is a UI driven model which plugs in the BizTalk Administration Console.

PAM defines 2 levels: Global and Party. Settings from the Global level are used during processing if no party is resolved for a document. R2 gives the users the ability to customize processing of individual documents by resolving them to a specific party. In other words, the global settings are enough to perform processing. Party specific properties are just an addition to the process that allows fine grain customization.

In addition to the levels, PAM defines 2 roles under Party settings: Party as Interchange Sender and Party as Interchange Receiver. Settings from Interchange Sender role are used during parsing and settings from Receiver role are used during serialization.

The following diagram explains the above

Roles

Global Settings:

The global properties are defined in a detailed post here.

Party Settings:

A party can be created by selecting the party node under BizTalk Server 2006 Administration Console and right clicking on the right hand view and selecting New->Party. Once a party is created, the EDI Party properties are accessed by right clicking on that party and selecting “EDI Properties“. The following snapshot shows the EDI properties view

EDI Properties

Party properties are divided into a General section used by all EDI components, a contact information tab and a separate EDIFACT and X12 section that is subdivided for Engine Receive, Send and Batching.

General Properties:

General properties include Activating Status Reporting which is turned off by default. It also has an option of copying settings from an already configured party. It also specifies certain agreement details like start and end date and a text field to add notes related to this agreement. General settings are applied for all EDI runtime components (except AS2 that has a separate page dedicated to it).

Contact Information:

Allows you to enter the contact information for the party. It includes fields like Name, Company and email etc.

EDIFACT Properties:

EDIFACT properties are used by the EDI runtime to process an EDIFACT encoded interchange. It is described in great detail here.

X12 Properties:

X12 properties are used by the EDI runtime to process X12 encoded interchange. It is described in great detail here.

Validation:

Validation of certain fields specified in PAM are done according to the standard defined by X12 and EDIFACT. The following rules are followed:

  • Validation of ISA, GS, UNB and UNG fields are done against the control schema deployed under the BizTalk EDI Application. These include checking for min/max length and performing data type validation.
  • Validation of delimiters is done to ensure the delimiters do not collide with other delimiters and they are under the valid range of 0-127 ASCII
  • For EDIFACT, UNB2.1, UNB2.2, UNB3.1 and UNB3.2 are validated as a quartet to ensure they are unique among all parties. The same applies for ISA5-8
  • For control numbers (ISA13, GS6, ST2, UNB5, UNG5 and UNH1), validation is performed to ensure their length is within the allowed range. Invalid values are not saved. If during processing of a doc the control numbers exceed maximum length, an error is raised in the event viewer and the document is suspended. A user has to manually reset the control number to allow further processing

End Note:

I hope this post has been useful in understanding PAM and how it is used by the runtime components during processing of an EDI document. Your comments/questions are always welcome.

Thanks

Mohsin Kalam

Advertisements

October 5, 2007

R2 Engine Send

Filed under: R2 EDI — Mohsin Kalam @ 11:59 am

Goal:

To provide in depth analysis of how R2 Engine Send serializes an EDI document. 

Overview:

Engine Send is implemented as a BizTalk pipeline with EDI Assembler component in the Assemble stage.

Engine Send:

Below is a diagram of the EDI Send Pipeline showing the individual component

EDI Send

EDI Send pipeline does the following when serializing a document

  • Party Resolution
  • Schema Discovery
  • Document Processing
  • Envelope Application
  • Apply Party Specific Properties (or Global)
  • EDI Validation
  • Character Set Validation
  • Status Reporting

Party Resolution:

Like Engine Receive, Engine Send requires party or global settings to apply during serialization. Party resolution is explained in detail here.

Schema Discovery:

Once the source of the settings is determined, Engine Send tries to discover the Schema. Schema discovery process is defined here.

Document Processing:

Like Receive, Engine Send also supports 3 different mode of processing but unlike receive where you have to configure the mode of processing, the processing mode in Engine Send is dynamic. They are as follows:

1. Serializing an Individual Transaction Set:

In this scenario, XML documents without the envelope segments are fed to the send pipeline and an EDI document is produced. The envelope information comes from Party as Interchange Receiver settings of either the party the document resolves to or global settings.

2. Serializing a BIBO Message:

In this scenario, the whole interchange with all the envelopes in XML format is fed to the send pipeline and the it is serialized. Since the envelope information and the delimiters are already present in the XML file so no additional settings are required.

3. Serializing a Batch Message:

This is essentially the same as BIBO but Send Pipeline updates selected fields in the envelopes. Send pipeline differentiates between the mode 2 and 3 by the presence of the promoted property “PopulateInterchangeValues” in a batch message. If this property is promoted and set to true, Send Pipeline updates selected fields (explained in the next section).

The difference between Batch and BIBO message is that a Batch message is Party dependent. Meaning if a party cannot be resolved to fetch the settings, the message will be suspended.

Envelope Application:

Envelope application is at the heart of Send Pipeline and the functionality varies for the 3 modes of processing. It is explained in great detail here.

Apply Party Specific Properties (or Global):

During serialization, R2 provides the ability to apply certain settings on the document being processed. These include generating trailing separators and turning validation on or off. If a party cannot be resolved, these settings are applied from a combination of global and pipeline component level settings. If a document resolves to a party, the settings are taken from that party. In other words, the global and pipeline settings are enough to perform processing. Party specific properties are just an addition to the process that allows fine grain customization. 

A separate post will follow that will explain all the properties in PAM, Global and Pipeline component settings.

EDI Validation:

EDI validation includes Min/Max length validation, Min/Max Occurs validation and validation of specific data types against the standard (e.g validating X12_Nn datatype only contains numeric type etc). A document failing any of these validation will be suspended.

In case of a party not being resolved, validation is always performed. At the party level, the default setting is to enable validation. The validation can be made lax by unchecking this option in PAM under Party as Interchange Receiver/ACK Processing and Validation Settings.

Please note that in case of BIBO or Batch processing mode, the envelope validation will always happen against the control schemas regardless of this option configured in PAM.

Character Set Validation:

EDI documents allow certain encoding types as defined by the specific standard. Each encoding type allow only certain characters as the valid characters. The engine performs this character set validation to ensure the characters fall within the allows range as the standard defines them. The detailed information can be found here.

Status Reporting:

EDI engine provided detailed status reporting for the documents flowing within the EDI system. The reporting is available at both the Party and Global level. Status reporting is turned off by default. When turned on, the status reporting provides details about Interchange and correlated ACK status, Batch status, Interchange Aggregation status for each party and Transaction Set Aggregation status for each Transaction set. The report is available within the group hub page under EDI Status Reports.

End Note:

I hope this post has been useful in understanding the EDI Engine Send. Your comments/questions are always welcome.

Thanks

Mohsin Kalam

October 2, 2007

R2 Engine Receive

Filed under: R2 EDI — Mohsin Kalam @ 5:06 pm

Goal:

To provide an in depth analysis of how R2 Engine Receive disassembles an EDI document. 

Overview:

Engine Receive is implemented as a BizTalk pipeline with EDI Disassembler component in the Disassemble stage and Batch Marker component in the Resolve Party stage. Batch Marker component is connected to the Batching sub-system and will be explained in another post in great detail.

Engine Receive:

Below is a diagram of the EDI Receive Pipeline showing the individual components

EDI Receive

EDI Receive pipeline does the following when parsing a document

  • Delimiter Discovery
  • Party Resolution
  • Performs Authentication
  • Schema Discovery
  • Document Processing
  • Apply Party Specific Properties (or Global)
  • EDI Validation
  • Character Set Validation
  • Status Reporting

Delimiter Discovery:

EDI files are flat files with delimiters separating the data so the first step in parsing is to determine these delimiters. Delimiter discovery in R2 is done on the fly and is explained in detail here.

Party Resolution:

Once the delimtiers are discovered, R2 engine parses the document based on the delimiters read to determine the party resolving to this document. This is needed to determine the source of settings to apply during parsing. Party resolution is explained in detail here.

Performs Authentication:

EDI Receive performs authentication on an incoming message at the Party level based on values specified in PAM. For X12, it matches the fields in ISA1-4 of the instance to the values in PAM and on mismatch, the whole interchange is suspended. For EDIFACT, it matches the fields in UNB6.1 and UNB6.2.

Authentication is only performed at the party level. If a party cannot be resolved, no authentication is performed.

Schema Discovery:

Once the source of the settings is determined, Engine Receive tries to discover the schema. Schema discovery process is defined here.

Document Processing:

With the delimiters discovered, the party resolved and the schema discovered, the Engine is ready to Parse the interchange to XML. In case the party is resolved, Engine Receive can be configured to supports 3 different modes of processing. These can be set under Party as Interchange Sender/Ack Generation and Validation Settings/Inbound Batch Processing option.

1. Split Interchange as Transaction Sets:

This is the default mode. As it suggests, the whole interchange is splitted into individual XML documents during parsing. For example if an interchange has 1 group with 3 transaction sets within that group, after parsing, the receive pipeline will produce 3 XML documents from the 3 transaction sets and promote certain properties on each of those documents. In case of validation errors in the envelope, the whole interchange will be suspended and an entry written in the event viewer. In case of validation errors in the individual transaction sets, only those transaction sets will be suspended.

2. Preserve Interchange – suspend Interchange on Error:

This is described as Batch In Batch Out (BIBO) scenario and is primarily used to validate the interchange as a whole. The whole interchange along with all the envelope information is transformed into XML. In case of validation errors in either the envelope of the individual transaction sets, the whole interchange will be suspended and an entry written in the event viewer.

3. Preserve Interchange – suspend Transaction Sets on Error:

This is BIBO scenario with the added clean up functionality. The whole interchange along with all the envelope information is transformed into XML. In case of validation errors in the envelope, the whole interchange will be suspended and an entry written in the event viewer. However in case of validation errors in the individual transaction sets, those transaction sets will be removed from the interchange and the rest of the document is preserved. So this is option gives the customer the option of preserving the whole interchange while cleaning up the faulty transaction sets.

Note: If a party cannot be resolved, then the choice of selecting the processing mode is not available and the default mode is implicitly applied!

Apply Party Specific Properties (Or Global):

During parsing, R2 provides the ability to apply certain settings on the document being processed. These include generating both technical and functional acknowledgements, specifying the mode of processing and turning validation on or off. If a party cannot be resolved, these settings are applied from a combination of global and pipeline component level settings. If a document resolves to a party, the settings are taken from that party. In other words, the global and pipeline settings are enough to perform processing. Party specific properties are just an addition to the process that allows fine grain customization. 

A separate post will follow that will explain all the properties in PAM, Global and Pipeline component settings.

EDI Validation:

EDI validation includes Min/Max length validation, Min/Max Occurs validation and validation of specific data types against the standard (e.g validating X12_Nn datatype only contains numeric type etc). A document failing any of these validation will be suspended.

In case of a party not being resolved, validation is always performed. At the party level, the default setting is to enable validation. The validation can be made lax by unchecking this option in PAM under Party as Interchange Sender/ACK Generation and Validation Settings.

Please note that the envelope validation will always happen against the control schemas regardless of this option configured in PAM.

Character Set Validation:

EDI documents allow certain encoding types as defined by the specific standard. Each encoding type allow only certain characters as the valid characters. The engine performs this character set validation to ensure the characters fall within the allows range as the standard defines them. The detailed information can be found here.

Status Reporting:

EDI engine provided detailed status reporting for the documents flowing within the EDI system. The reporting is available at both the Party and Global level. Status reporting is turned off by default. When turned on, the status reporting provides details about Interchange and correlated ACK status, Batch status, Interchange Aggregation status for each party and Transaction Set Aggregation status for each Transaction set. The report is available within the group hub page under EDI Status Reports.

End Note:

I hope this post has been useful in understanding the EDI Engine Receive. Your comments/questions are always welcome.

Thanks

Mohsin Kalam

September 27, 2007

Introduction to Microsoft BizTalk Server 2006 R2 EDI

Filed under: R2 EDI — Mohsin Kalam @ 3:21 pm

Goal:

The goal of this posting is to provide an overview of EDI and how it is implemented in Microsoft BizTalk Server 2006 R2. This will be followed by a detailed posting on the underlying subsystems involved.

Terminologies:

The following terms are used within this blog to mean things specific to EDI R2. These will be used in the future to maintain consistency.

  1. Parsing: Conversion of EDI document to XML
  2. Serialization: Conversion of XML document to EDI 
  3. PAM: Partner Agreement Manager
  4. BizTalk, R2, R2 EDI: They all refer to Microsoft BizTalk Server 2006 R2

Overview: 

Microsoft BizTalk Server 2006 R2, the latest release of BizTalk Server, features broad support of WS* protocols through its WCF adapters, support for integration via Microsoft BizTalk RFID, and support for integrating business partners though EDI and AS2. These new features enhance an already robust product and deliver a high value proposition for the customers making BizTalk Server a great choice for Business to Business integration.

Since this blog is geared towards EDI, I would not go in any detail about WCF and RFID. 

EDI Summary:

Electronic Data Interchange (EDI) is a standard based messaging protocol and is the most prevalent means by which business entities communicate and exchange data electronically. A quick snapshot will include majority of Fortune 500 and other well known global companies like Walmart, GE, American Airlines, Intel, Nike and JC Penney etc.

EDI is an industry agnostic standard and is reported to be widely used in manufacuring, shipping, warehousing, construction, food processing and healthcare.

EDI data is transmitted as delimited files (without self describing tags) and therefore the encoding rules enforce very strict formatting rules to ensure the destination application is able to succesfully parse and consume the information.

There are 2 major versions of EDI, X12 and EDIFACT. X12 is used in North America whereas EDIFACT is used in Europe and Asian markets. R2 supports both these versions and ships with 8000+ schemas out of box for these standards. For a complete list of supported versions, please refer to this link.

R2 System Architecture Overview:

R2 EDI is a major upgrade to its predecessor. All the EDI logic has been moved out of the EDI specific processing layer and into BizTalk itself. This is achieved by creating Receive and Send Pipeline Components (that reside in Receive and Send Pipelines respectively) that can parse and serialize EDI messages. In addition to that R2 provides support for aggregating documents together (Batching), AS2, detailed Status Reporting and Design Time user experience.

Below is a high level architecture for R2 EDI

R2 Architecture

EDI Receive:  

EDI Receive and EDI Send form the EDI Engine. The EDI Receive Engine is responsible for parsing an incoming message, apply settings during parsing and create status reporting for tracking the documents flowing through the EDI system.  

Detailed information explaining EDI Engine Receive and its features can be found in another posting here.

EDI Send:

The EDI Send Engine is responsible for serializing an outgoing message, create envelope information, apply settings during serialization and create status reporting. Send can be considered the counterpart for Receive because it converts XML to EDI where as Receive converts EDI to XML.

Detailed information explaining EDI Engine Send and its features can be found in another posting here.

Batching Sub-System:

R2 provides out of box support for aggregating (batching) transaction sets in an interchange. This is accomplished by the use of Batch Marker component in the Receive Pipeline and Batching and Routing Orchestrations.  

Detailed information explaining EDI Batching and its features can be found in another posting here.

PAM:

PAM is the central repository for storing all the settings required by the EDI Engine and Batching Sub-System during processing. It has a clean UI that plugs in the BizTalk administration console providing a single point for tweaking all EDI settings used by R2.

Detailed information explaining PAM and its features can be found in another posting here.

Status Reporting:

R2 provides detailed reporting functionality to track business documents as they travel within R2. This is nicely integrated within the BizTalk administrative console providing a single point of tracking for all EDI related documents.

Design Time User Experience:

R2 provides a design time experience for users to Validate/Generate an interchange or a transaction set and Validate a schema. This experience is hosted within Visual Studio and provides the ability for users to validate a document to find out the errors and fix those errors in the instance.

Detailed information explaining Design Time user experience and its features can be found in another posting here.

AS2 Support:

Microsoft BizTalk R2 has built in support for the Applicability Statement (AS2) out of box. This is achieved by providing 4 pipelines: AS2 Receive, AS2 EDI Receive, AS2 Send and AS2 EDI Send. AS2 Receive has AS2 decoder in the decode stage of the Receive pipeline where as its counterpart AS2 Send has AS2 encoder in the encode stage of the Send pipeline.

AS2 is payload agnostic and is used as an alternative to value added networks (VANs). The use of internet for data exchange, instead of direct point-to-point connections, reduces costs, increases flexibility and efficiency, and has advantages in terms of redundancy and scalability.

BizTalk Server 2006 R2 uses AS2-defined methods to send, receive, encrypt, decrypt, sign, and verify messages between partners using HTTP over the Internet. BizTalk Server helps ensure the security of messages through the use of encryption keys, digital signatures, certificates, and non-repudiation.   

End Note: 

I hope this posting has been informative for you. Please keep tuned for the next postings that describe each section in detail. And please feel free to leave comments/questions.

Thanks

Mohsin Kalam

September 17, 2007

Welcome to Mohsinkalam’s Blog

Filed under: R2 EDI — Mohsin Kalam @ 10:28 am

Hello everyone!

I am Mohsin Kalam and I am a Software Design Engineer in Test in the Microsoft BizTalk Server EDI team. 

This blog is set up to focus on the new Microsoft BizTalk Server R2 support of B2B transactions while focusing primarily on EDI. It is also intended to provide a channel to reach our customers, partners and the user community. 

Please feel free to leave your comments/suggestions, ask clarifying questions or let me know if you would like me to blog about a certain topic in R2 EDI.

Thanks for stopping by. Have a great day!

Mohsin Kalam

« Previous Page

Create a free website or blog at WordPress.com.