Streaming Access Logs to Amazon Kinesis

Foxpass can send LDAP and RADIUS logs to your Amazon Web Services account in real time. Foxpass will send logs to the Amazon Kinesis data streaming service which can be connected to different data processing engines. For example, you can:

To use this feature, you or your company must have an Amazon Web Services account and must have purchased the Foxpass advanced logging add-on. You will be responsible for all AWS data processing and storage charges incurred by use of this feature.

Amazon Kinesis record format

Foxpass logs each authentication request to your Kinesis stream as a separate record. Each record is formatted a JSON data structure with a newline (\n) at the end. The JSON data structure differs depending on the service being logged, but always contains the common fields.

Common fields

  • service: always "ldap" for LDAP authentication requests, "radius" for RADIUS authorization requests
  • method: the action that was attempted; presently "bind" is the only method logged for LDAP, and "authorize" is the only method logged for RADIUS
  • message: a string describing the event being logged
  • timestamp: the time that the action was attempted, formatted like "YYYY-MM-DD HH:MM:SS.sss" (for example, "2018-01-15 14:29:38.175")
  • success: a boolean indicating whether the request succeeded

Fields for LDAP logs

  • bind_dn: the distinguished name used in the LDAP bind attempt
  • user_type: "user" for normal users, "binder" for LDAP binders, or null if the user was not found
  • username: username of the user who attempted to authenticate, or null if the user was not found

Fields for RADIUS logs

  • ipAddress: the IP address that was the source of the RADIUS authorization request
  • username: username of the user who attempted to authenticate, or null if the user was not found

Automatically configuring a Kinesis stream in AWS

Foxpass offers two Amazon CloudFormation templates to automate the creation of AWS resources for this feature. These templates will create a new CloudFormation stack, which is a collection of AWS resources managed as a single unit.

  • Template 1: Publish logs to a Kinesis stream: With this template, Foxpass will send log entries to an Amazon Kinesis stream in your AWS account, at which point you can connect other software to the Kinesis stream. This configuration will also create the IAM role that allows Foxpass to access your Kinesis stream. This configuration is appropriate for pushing Foxpass logs to Kinesis-enabled utilities like SumoLogic (instructions), Logstash (instructions), or Splunk (instructions). This can also be used with custom consumer code, either using AWS Lambda (instructions) or the Amazon Kinesis API.

  • Template 2: Store logs in Amazon S3: Using this template, Foxpass will send log entries to an Amazon Kinesis stream in your AWS account. This configuration is appropriate for long-term archival of Foxpass logs, or for loading the logs into another system that can read from Amazon S3. In this configuration, a new log file is written to S3 about once a minute. The log file contains one record per line, with each record stored as JSON. Logs are retained permanently, but you could add a lifecycle policy to apply a more limited retention policy.

📘

Tips on working with S3 buckets

S3 bucket names are unique across all existing bucket names in S3, so you’ll need to provide a unique name during creation to avoid conflicting with an existing bucket.

If you want to store Foxpass logs in an existing S3 bucket, you’ll need to modify the CloudFormation template to use that bucket rather than creating a new one.

Applying a CloudFormation template to your AWS account

  1. Login to the AWS Console and navigate to the CloudFormation Management Console.
  2. Click the “Create Stack” button to begin creating a new CloudFormation stack.
  3. On the next page, choose “Specify an Amazon S3 Template URL” and provide the right URL:
  4. On the following page you will be asked to provide a name for the stack, and the name of the Amazon Kinesis stream or S3 bucket to be created.
  5. The next page will show you a number of advanced CloudFormation options. You don’t need to change any of those options.
  6. The following step will show you a summary of the options you selected. To begin creating the AWS resources used by Foxpass, click “Create” at the bottom of the page.
  7. This will take you to the list of CloudFormation stacks in your account. The stack you just initiated will show its state as CREATE_IN_PROGRESS. Wait for it to change to status CREATE_COMPLETE.
  8. Once the stack is ready, click on the newly-created stack in the list, and click on the “Outputs” tab. Make a note of these four values as you’ll need them to configure Foxpass. Skip ahead to “Configuring Foxpass to send logs to Kinesis.”

Manually configuring a Kinesis stream in AWS

If you’d like more control over the AWS resources used by Foxpass, you can manually configure an Amazon Kinesis stream and an IAM role to be used by Foxpass. At runtime, Foxpass will use this IAM role to connect to your Kinesis stream.

Kinesis stream configuration

You can create an Amazon Kinesis stream exclusively for Foxpass logs, or Foxpass can write to an existing Amazon Kinesis stream in your account. No special configuration is necessary for the stream. If the stream has more than one shard, Foxpass log records will be spread across all shards equally and without any regard to users or operations (that is, each record will have a random Partition Key).

IAM role configuration

You will also need to create an IAM role that allows Foxpass to write to your Kinesis stream. This role should be assumable by AWS account ID 843529240033 and must require an External ID (but not multi-factor authorization). The external ID to use is generated by Foxpass when entering the Kinesis details in the console (Click the Add Logging Configuration button on the Kinesis Logging page to being the process). Here is an example of the trust relationship policy document that allows Foxpass to assume the role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::843529240033:root"
      },
      "Condition": {"StringEquals": {"sts:ExternalId": "<ExternalID Assigned by Foxpass>"}},
      "Action": "sts:AssumeRole"
    }
  ]
}

The role should have an IAM policy that allows three actions on your Kinesis stream: kinesis:DescribeStream, kinesis:PutRecord, and kinesis:PutRecords. Here is an example of that IAM policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kinesis:DescribeStream",
                "kinesis:PutRecord",
                "kinesis:PutRecords"
            ],
            "Resource": "<stream ARN>"
        }
    ]
}

Configuring Foxpass to send logs to Kinesis

Foxpass customers with the Event Logging add-on will see a page in the Foxpass console called Kinesis Logging. This page lets you set up one or more Kinesis streams which will receive records for all LDAP and RADIUS authentication events within your Foxpass account.

🚧

Tip: It will take up to one minute for Kinesis Logging config changes to take effect.

Configuring Foxpass to write to a new Kinesis stream

You will need 4 bits of information to create a Kinesis logging configuration in Foxpass. If you have created the AWS resources with one of the CloudFormation templates above, all of the settings are available in your CloudFormation stack’s “Outputs” tab.

  • AWS account ID: Your twelve-digit Amazon account ID
  • AWS region: The name of the AWS region where your Kinesis stream is located
  • IAM role name: The name of the AWS IAM role that Foxpass is allowed to assume
  • Kinesis stream name: The name of the Kinesis stream that Foxpass should write to

To configure Foxpass to write to a new stream, navigate to the Kinesis Logging page in the Foxpass console, and click “Add Logging Configuration.” Enter a freeform descriptive name for the configuration, as well as the four pieces of information described above. When you click “Test & Add,” Foxpass will attempt to assume the IAM role and access the Kinesis stream, and it will be saved once the test passes.

Enabling, disabling, and deleting logging configurations

You can disable a logging configuration by choosing “Disable config” from the “Actions” menu next to each logging configuration. You can also delete an existing configuration by choosing “Delete config” from the “Actions” menu.