Skip to content
This repository has been archived by the owner on Dec 7, 2023. It is now read-only.

Several Docs Missing in Repo, but Referenced and Published on Live AWS User Guide? #428

Closed
ScriptAutomate opened this issue Aug 14, 2019 · 9 comments
Assignees
Labels

Comments

@ScriptAutomate
Copy link

I've been auto-generating code from CFN Resource specification files, and part of the generation runs through and automatically builds help documentation. Though, many of the source files that I would be pulling from only exist on the live user guide and not in the GitHub source documentation. Is there some kind of release strategy where documentation becomes available in the official AWS documentation before hitting the source repo here?

Existing Public Page Missing from GitHub
AWS::CodeBuild::SourceCredential aws-resource-codebuild-sourcecredential.md
AWS::CodeStar::GitHubRepository aws-resource-codestar-githubrepository.md
AWS::Glue::MLTransform aws-resource-glue-mltransform.md
AWS::LakeFormation::DataLakeSettings aws-resource-lakeformation-datalakesettings.md
AWS::LakeFormation::Permissions aws-resource-lakeformation-permissions.md
AWS::LakeFormation::Resource aws-resource-lakeformation-resource.md
AWS::Batch::JobDefinition Device aws-properties-batch-jobdefinition-device.md
AWS::Batch::JobDefinition LinuxParameters aws-properties-batch-jobdefinition-containerproperties-linuxparameters.md
AWS::CodeStar::GitHubRepository Code aws-properties-codestar-githubrepository-code.md
AWS::CodeStar::GitHubRepository S3 aws-properties-codestar-githubrepository-s3.md
AWS::Cognito::UserPool UserPoolAddOns aws-properties-cognito-userpool-userpooladdons.md
AWS::Cognito::UserPool VerificationMessageTemplate aws-properties-cognito-userpool-verificationmessagetemplate.md
AWS::Cognito::UserPoolClient AnalyticsConfiguration aws-properties-cognito-userpoolclient-analyticsconfiguration.md
AWS::DLM::LifecyclePolicy Parameters aws-properties-dlm-lifecyclepolicy-parameters.md
AWS::EFS::FileSystem LifecyclePolicy aws-properties-elasticfilesystem-filesystem-lifecyclepolicy.md
AWS::Glue::MLTransform FindMatchesParameters aws-properties-glue-mltransform-transformparameters-findmatchesparameters.md
AWS::Glue::MLTransform GlueTables aws-properties-glue-mltransform-inputrecordtables-gluetables.md
AWS::Glue::MLTransform InputRecordTables aws-properties-glue-mltransform-inputrecordtables.md
AWS::Glue::MLTransform TransformParameters aws-properties-glue-mltransform-transformparameters.md
AWS::LakeFormation::DataLakeSettings Admins aws-properties-lakeformation-datalakesettings-admins.md
AWS::LakeFormation::DataLakeSettings DataLakePrincipal aws-properties-lakeformation-datalakesettings-datalakeprincipal.md
AWS::LakeFormation::Permissions DatabaseResource aws-properties-lakeformation-permissions-databaseresource.md
AWS::LakeFormation::Permissions DataLakePrincipal aws-properties-lakeformation-permissions-datalakeprincipal.md
AWS::LakeFormation::Permissions Resource aws-properties-lakeformation-permissions-resource.md
AWS::LakeFormation::Permissions TableResource aws-properties-lakeformation-permissions-tableresource.md
@AWSRandall
Copy link
Contributor

Hey, thanks for bringing up the issue! Your script sounds cool. I'd be interested to know whether you also generate the content portions (outside of the structures and parameters).

We also auto-generate our documentation from the CFN resource specification. As such there is a slight delay before we update our GitHub docs repo with the latest supported resources and properties (see the Release History here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/ReleaseHistory.html). We try to keep the delay as short as possible so there's very little drift.

Having said all that, we'll double check your diff to see if these all fall into that category of recently added resources/properties. Thanks for pointing this out!

@ScriptAutomate
Copy link
Author

Hey, thanks for bringing up the issue! Your script sounds cool. I'd be interested to know whether you also generate the content portions (outside of the structures and parameters).

We also auto-generate our documentation from the CFN resource specification. As such there is a slight delay before we update our GitHub docs repo with the latest supported resources and properties (see the Release History here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/ReleaseHistory.html). We try to keep the delay as short as possible so there's very little drift.

Ah, cool, that helps explain. I completely overlooked the Release History page! I'll definitely have some comparison lookups in the future to see if missing stuff is likeley new stuff on delay. I should have also said this is using the latest us-west-2 cfn resource spec file, as I haven't included eval of the other regions.

I do actually pull other content to add into my code, for documentation purposes, such as resource and parameter descriptions. The current example is where I am using CFN resource documentation to generate resource-specific PowerShell Core functions. Here is an example output of one:

function New-AWSCFNApiGatewayAccount {
<#
.Synopsis
The AWS::ApiGateway::Account resource specifies the IAM role that Amazon API Gateway uses to write API logs to Amazon CloudWatch Logs.
.DESCRIPTION
The AWS::ApiGateway::Account resource specifies the IAM role that Amazon API Gateway uses to write API logs to Amazon CloudWatch Logs.
**Important**
If an API Gateway resource has never been created in your AWS account, you must add a dependency on another API Gateway resource, such as an AWS::ApiGateway::RestApi: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigateway-restapi.html or AWS::ApiGateway::ApiKey: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigateway-apikey.html resource.
If an API Gateway resource has been created in your AWS account, no dependency is required (even if the resource was deleted.
 .PARAMETER CloudWatchRoleArn
The Amazon Resource Name (ARN of an IAM role that has write access to CloudWatch Logs in your account.
 .LINK
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigateway-account.html
 #>
  [CmdletBinding()]
  param(
    [Parameter(Mandatory=$False)]
    [String]$CloudWatchRoleArn
  )
  [PSCustomObject]@{
    'Type' = 'AWS::ApiGateway::Account'
    'Properties' = @{
      'CloudWatchRoleArn' = "$CloudWatchRoleArn"
    }
  }
}

PowerShell Core comment-based help allows for help documentation in-terminal output to look like this:

PS /home/scriptautomate> Get-Help New-AWSCFNApiGatewayAccount -Full  

NAME
    New-AWSCFNApiGatewayAccount
    
SYNOPSIS
    The AWS::ApiGateway::Account resource specifies the IAM role that Amazon API Gateway uses to write API logs to Amazon CloudWatch Logs.
    
    
SYNTAX
    New-AWSCFNApiGatewayAccount [[-CloudWatchRoleArn] <String>] [<CommonParameters>]
    
    
DESCRIPTION
    The AWS::ApiGateway::Account resource specifies the IAM role that Amazon API Gateway uses to write API logs to Amazon CloudWatch Logs.
    **Important**
    If an API Gateway resource has never been created in your AWS account, you must add a dependency on another API Gateway resource, such 
    as an AWS::ApiGateway::RestApi: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigateway-restapi.html or 
    AWS::ApiGateway::ApiKey: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigateway-apikey.html resource.
    If an API Gateway resource has been created in your AWS account, no dependency is required (even if the resource was deleted.
    

PARAMETERS
    -CloudWatchRoleArn <String>
        The Amazon Resource Name (ARN of an IAM role that has write access to CloudWatch Logs in your account.
        
        Required?                    false
        Position?                    1
        Default value                
        Accept pipeline input?       false
        Accept wildcard characters?  false
        
    <CommonParameters>
        This cmdlet supports the common parameters: Verbose, Debug,
        ErrorAction, ErrorVariable, WarningAction, WarningVariable,
        OutBuffer, PipelineVariable, and OutVariable. For more information, see
        about_CommonParameters (https://go.microsoft.com/fwlink/?LinkID=113216). 
    
INPUTS
    
OUTPUTS
    
    
RELATED LINKS
    http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigateway-account.html

With the ability to convert into JSON:

PS /home/scriptautomate> New-AWSCFNApiGatewayAccount -CloudWatchRoleArn 'arn.xxxxx.xxx' | ConvertTo-Json
{
  "Type": "AWS::ApiGateway::Account",
  "Properties": {
    "CloudWatchRoleArn": "arn.xxxxx.xxx"
  }
}

It's a work in progress, but being able to simply pull this git repo, and parse the appropriate Markdown docs allows for easy built-in documentation/help generation. I'm going to be extending it to include region-specific rules/checks, so I had to build a tool that will create and manage my own reference repository as I couldn't find the spec files in any AWS github repo: ScriptAutomate/aws-cfn-resource-specs (I point this out in #4 )

Broken Documentation Links (using us-west-2 spec file as reference)

Does the lag in source repo documentation replication explain these Broken Documentation Links in CloudFormation Resource Specification Files? These links include for base of resource, and all sub-instances of link referenced by sub properties:

Resource BrokenLink WorkingLink
AWS::AutoScaling::AutoScalingGroup.InstancesDistribution Broke ReplaceWith
AWS::AutoScaling::AutoScalingGroup.MixedInstancesPolicy Broke ReplaceWith

If not, I'll open a second issue in relation to the broken links.

@PatMyron
Copy link
Contributor

@ScriptAutomate Resource specification for us-east-1 will have a bit more than us-west-2:

"region-specific rules/checks" is interesting, would cfn-lint be a good fit?

@ScriptAutomate
Copy link
Author

ScriptAutomate commented Sep 1, 2019

@PatMyron thanks for the info on most-populated resource specs, and I'll also be taking a look at cfn-lint!

@AWSRandall said:
We also auto-generate our documentation from the CFN resource specification. As such there is a slight delay before we update our GitHub docs repo with the latest supported resources and properties (see the Release History here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/ReleaseHistory.html). We try to keep the delay as short as possible so there's very little drift.

Looking at the release history over the past week, there haven't been any new resource specs that have been published reflecting the updates from the Aug. 29th update. Is there a similar lag time that occurs with spec json files reflecting changes referenced in release version history? For example, AWS::Config::OrganizationConfigRule was recently added as a new Config Resource Type, but doesn't appear in any region's cfn resource spec json (v5.3.0 is latest as of Aug. 31).

Having said all that, we'll double check your diff to see if these all fall into that category of recently added resources/properties. Thanks for pointing this out!

I haven't done a new Documentation link scan since opening this issue, so I don't know if any new doc links may have been added that are to documentation not with the source repo. With that being said, and reviewing the table in this issue, I believe the GitHub source documentation has been fully caught up with what I found missing or broken except for:

Existing Public Page Missing from GitHub Release History Date Ref
AWS::EFS::FileSystem LifecyclePolicy aws-properties-elasticfilesystem-filesystem-lifecyclepolicy.md June 13, 2019

Remaining stuff:

  • I'll leave this issue open in the meantime until the documentation is added here, since this doesn't seem to be a natural time gap related to lag time. I'd like to include a day count range in CFN spec resource file updates where, if the documentation has yet to appear in the GitHub source doc repo, it will wait x amount of days before an issue should be raised in this repository. I'll probably set a 1 - 2 week expectation in my tooling, so that I raise an issue only if any documentation is still lagging behind. Would an expectation of that length be reasonable?
  • Should I open a new issue for the broken Documentation links referenced in the CFN resource spec files? As in, the links that are present, but have since moved or were otherwise incorrect URL links? I am only asking because I do not know where else to raise this issue, since the CFN resource files themselves are not versioned within this AWS repo and this documentation repo seemed the best bet (I created Missing CloudFormationResourceSpecification.json Files in This Repository #4 a while back for this reason). The latest CFN spec version (v5.3.0) still has the following broken Documentation links for these resources (total of 10 broken links in the spec):
Resource BrokenLink WorkingLink
AWS::AutoScaling::AutoScalingGroup.InstancesDistribution Broke ReplaceWith
AWS::AutoScaling::AutoScalingGroup.MixedInstancesPolicy Broke ReplaceWith

NOTE: I've created an issue in my own repository for these broken links where I am tracking these files at ScriptAutomate/aws-cfn-resource-specs: [v5.1.0] Broken Documentation Links

@ScriptAutomate
Copy link
Author

UPDATE: I have finished some automated tooling around broken Documentation field lookups in CFN resource spec files, as issue tracked here: [Documentation Tracker JSON] Verification of Documentation Links Working, and Existing in AWS CFN Doc Source Repo. Here are most-current results:

I found a total of 17 issues. They seem to be a mix of issues around:

  • GitHub AWS CFN UserGuide Doc source
  • AWS CFN UserGuide website
  • The CloudFormationResourceSpecification.json files
  • The process in which the GitHub source is updated to reflect the UserGuide website content

Here are the types of issues I've come across:

BrokenDocLinks

  • AWS Doc html page is broken, with this page unreachable and redirecting to UserGuide home. Seems to be an issue with the AWS CloudFormation User Guide website.
  • URL in Documentation key of CloudFormationResourceSpecification.json file is pointing to an incorrect html page, and needs to be updated. Seems to be an issue with the CloudFormationResourceSpecification.json file.

MissingDocLinks

  • Documentation key is not present in the CloudFormationResourceSpecification.json file, and should be added. Seems to be an issue with the CloudFormationResourceSpecification.json file.

MissingFromAWSGitHubSourceRepo

  • Source repo pages are missing, even though they are present in AWS CFN UserGuide docs. Both causes seem to point to the possibility of the GitHub repo being updated via a manual instead of automated process, allowing for unclear periods of documentation lag and the potential of missing content to go unsynced. The causes look to be either:
    • This source repository has yet to catch up to the latest release(s), as @AWSRandall has mentioned as possibility and has been the case for most links referenced in this open issue. Source repo expected to lag behind in content by maybe 1 - 2 weeks.
    • This source repository never had the documentation pushed to it, not merely because it is lagging behind, but was missed at some point. This seems to currently be the case for AWS::DLM::LifecyclePolicy.Parameters and AWS::EFS::FileSystem.LifecyclePolicy, both part of a collection of updates referenced in release history from a June 13, 2019 release.

@ScriptAutomate
Copy link
Author

ScriptAutomate commented Sep 30, 2019

11/11/2019 EDIT: Increased days of time in below table for latest 8.0.0 cfn spec file release.

10/31/2019 EDIT: Increased days of time where items are still broken by another 30 days.

Based on the history of this issue, and what is actually considered missing docs that aren't merely a result of the slight lag between source documentation syncs (as in, has the error existed for greater than 2 weeks, as mentioned by @AWSRandall ), these are the 14 missing or otherwise broken documentation errors spread across the three categories mentioned in the previous comment:

Resource DaysBrokenAtLeast Error
AWS::Transfer::User.SshPublicKey 94 MissingDocLinks
AWS::EC2::LaunchTemplate.CapacityReservationPreference 94 MissingDocLinks
AWS::AutoScaling::AutoScalingGroup.InstancesDistribution 94 BrokenDocLinks
AWS::AutoScaling::AutoScalingGroup.LaunchTemplate 94 BrokenDocLinks
AWS::AutoScaling::AutoScalingGroup.LaunchTemplateOverrides 94 BrokenDocLinks
AWS::AutoScaling::AutoScalingGroup.MixedInstancesPolicy 94 BrokenDocLinks
AWS::CodePipeline::CustomActionType.ArtifactDetails 80 BrokenDocLinks
AWS::CodePipeline::CustomActionType.ConfigurationProperties 80 BrokenDocLinks
AWS::CodePipeline::CustomActionType.Settings 80 BrokenDocLinks
AWS::OpsWorks::Instance.EbsBlockDevice 80 BrokenDocLinks
AWS::OpsWorks::Layer.VolumeConfiguration 80 BrokenDocLinks
AWS::DLM::LifecyclePolicy.Parameters 151 MissingFromAWSGitHubSourceRepo
AWS::EFS::FileSystem.LifecyclePolicy 151 MissingFromAWSGitHubSourceRepo
AWS::Glue::Trigger.NotificationProperty 45 MissingFromAWSGitHubSourceRepo

This table was generated from documentation-broken-links-detailed.json, located within my CFN auditing repository. That JSON includes potential fixes for most of the issues, and should be referenced when resolving problems along with the previous comment defining BrokenDocLinks, MissingDocLinks, and MissingFromAWSGitHubSourceRepo

@ScriptAutomate
Copy link
Author

I will no longer be updating the table in the last comment. The most up-to-date documentation lookup errors can be found here, which will continue keeping results of AWS documentation audits for changes/errors/etc:

It includes a list of errors that will be related to this issue. As of today, at least 10 or 13 of the documentation errors have existed for anywhere between 100 and 210 days (broken documentation links, completely missing documentation, absence of documentation link in spec file, missing in GitHub source mirror, etc.).

Some additional, helpful information (but not fully updated/accurate because it's rather tedious and no longer managed), can be found here:

If there are any questions, feel free to ask.

@ScriptAutomate
Copy link
Author

I have since archived the following project:

If this is of any assistance, the final generated list of broken links was last done back in January 2023, so I don't know how valid they may be anymore:

@joshbean
Copy link
Contributor

joshbean commented Dec 7, 2023

Closing this issue or pull request in advance of archiving this repo. For more information about the decision to archive this repo (and others in the 'awsdocs' org), see the announcement on the AWS News Blog.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

5 participants