los verdaderos reyes de la noche vampiros demonios o angeles de la noche

nefertiti

Automating REST Security Part 3: Practical Tests for Real-World APIs

If you have read our two previous blogposts, you should now have a good grasp on the structural components used in REST APIs and where there are automation potentials for security analysis. You've also learned about REST-Attacker, the analysis tool we implemented as a framework for automated analysis.

In our final blogpost, we will dive deeper into practical testing by looking at some of the automated analysis tests implemented in REST-Attacker. Particularly, we will focus on three test categories that are well-suited for automation. Additionally, we will look at test results we acquired, when we ran these tests on the real-world API implementation of the services GitHub, Gitlab, Microsoft, Spotify, YouTube, and Zoom.

Author

Christoph Heine

Overview

Undocumented Operations

The first test that we are going to look at is the search for undocumented operations. These encompass all operations that accessible to API clients despite not being listed in the API documentation. For public-facing APIs, undocumented operations are a security risk because they can expose functionality of the service that clients are not supposed to access. Consequences can range from information leakage to extensive modification or even destruction of the resources managed by the underlying service.

A good example for an operation that should not be available is write access to the product information of a webshop API. While read operations on stock amounts, prices, etc. of a product are perfectly fine, you probably don't want to give clients the ability to change said information.

In HTTP-based REST, operations are represented by the HTTP methods used in the API request (as explained in Part 1 of the blog series). Remember that API requests are essentially HTTP requests which consist of HTTP method (operation), URI path (resource address) and optional header or body data.

GET /api/shop/items

We can use the fact that REST operations are components from the HTTP standard to our advantage. First of all, we know that the set of possible operations is the same for all HTTP-based REST APIs (no matter their service-specific context) since each operation should map to a standardized HTTP method. As a result, we also have a rough idea what each operation does when it's applied to a resource, since it's based on the assigned purpose of the HTTP method. For example, we can infer that the DELETE method performs a destructive action a resource or that GET provides a form of read access. It also helps that in practice most APIs only use the same 4 or 5 HTTP methods representing the CRUD operations: GET, POST, PUT, PATCH, and DELETE.

If we know a URI path to a resource in the API, we can thus enumerate all possible API requests, simply by combining the URI with all possible HTTP methods:

GET    /api/shop/items POST   /api/shop/items PUT    /api/shop/items PATCH  /api/shop/items DELETE /api/shop/items

REST-Attacker's test case undocumented.TestAllowedHTTPMethod uses the same approach to find undocumented operations. With an OpenAPI description, the generation of API requests is extremely to automate as the description lists all defined URI paths. Since the API description also documents the officially supported operations, we can slightly optimize the search by only generating API requests for operations not documented for a path (which basically are the candicates for undocumented operations).

To find out whether an undocumented operation exist, we have to determine if the generated API requests are successful. Here, we can again rely on a standard HTTP components that are used across REST APIs. By checking the HTTP response code of the API, we can see whether the API request was rejected or accepted. Since the response codes are standardized like the HTTP methods, we can also make general assumptions based on the response code received. If the operation in the API request is not available, we would expect to get the dedicated response code 405 - Method Not Allowed in the response. Other 4XX response codes can also indicate that the API request was unsuccessful for other reasons. If the operation is accepted, we would expect the API response to contain a 2XX response code.

Using the same approach, we let REST-Attacker search for undocumented operations in all 6 APIs we tested. None of them exposed undocumented operations that could be identified by the tool, which means they would be considered safe in regards to this test. However, it's interesting to see that the APIs could responded very differently to the API requests sent by the tool, especially when considering the response codes.

API	Response Codes
GitHub	`401`, `404`
Gitlab	`400`, `404`
MS Graph	`400`, `401`, `403`, `404`
Spotify	`405`
YouTube	`404`
Zoom	`400`, `401`, `403`, `404`, `405`

Spotify's API was the only one that used the 405 response code consistently. Other APIs returned 400, 401, 403, or 404, sometimes depending on the path used in the the API request. It should be noted that the APIs returned 401 - Unauthorized or 403 - Forbidden response codes even when supplying credentials with the highest possible level of authorization. An explanation for this behaviour could be that the internal access checks of the APIs work differently. Instead of checking whether an operation on a resource is allowed, they may check whether the client sending the request is authorized to access the resource.

Credentials Exposure

Excessive Data Exposure from OWASP's Top 10 API Security Issues is concerned with harmful "verbosity" of APIs. In other words, it describes a problem where API responses contain more information than they should return (hence excessive exposure). Examples for excessive data exposure include leaks of private user data, confidential data about the underlying service, or security parameters of the API. What counts as excessive exposure can also depend on the application context of the underlying service.

Since the definition of excessive data exposure is very broad, we will focus on a particular type of data for our practical test: Credentials. Not only do credentials exist in some form for almost any service, their exposure would also have a significant impact on the security of the API and its underlying service. Exposed credentials may be used to gain higher privileges or even account takeovers. Therefore, they are a lucrative target for attacks.

There are several credential types that can be interesting for attackers. Generally, they fit into these categories:

long-term credentials (e.g., passwords)
short-term credentials (e.g., session IDs, OAuth2 tokens)
service-specific credentials for user content (e.g., passwords for files on a file-hosting service)

Long- and short-term credentials should probably never be returned under any circumstances. Service-specific credentials may be less problematic in some specific circumstances, but should still be handled with care as they could be used to access resources that would otherwise be inaccessible to an API client.

The question is: Where can we start looking for exposed credentials? Since they would be part of the API responses, we could scrape the parameters in the response content. However, we may not actually need to look at any response values. Instead, we can examine the parameter names and check for association with credentials. For example, a parameter names "password" would likely contain a type of credential. The reason this can work is that parameter names in APIs are generally descriptive and human-readable, a side effect of APIs often being intended to be used by (third-party) developers.

In REST-Attacker, credentials parameter search is implemented by the resources.FindSecurityParameters test case. The test case actually only implements an offline search using the OpenAPI description, as the response parameter names can also be found there. The implementation iterates through the response parameter names of each API endpoint and matches them to keywords associated with credentials such as "pass", "auth" or "token". This naive approach is not very accurate and can produce a number of false-positives, so the resulting list of parameters has to be manually checked. However, the number of candidates is usually small enough to be searched in a small amount of time, even if the API defines thousands of unique response parameters.

API	Parameter Count	Candidates	service-specific
GitHub	2110	39	0
Gitlab	1291	0	0
MS Graph	32199	117	0
Spotify	290	6	0
YouTube	703	6	0
Zoom	800	96	2

5 out of 6 APIs we tested had no problems with exposed credentials.

Zoom's API was the only one which showed signs of problematic exposure of service-specific credentials by returning the default meeting password for meetings created via the API at an endpoint. It should be noted that this information was only available to approved clients and an required authorized API request. However, the credentials could be requested with few priviledges. Another problem was that Zoom did not notify users that this type of information was accessible to third-party clients.

Default Access Priviledges

The last test category that we are going to look at addresses the access control mechanisms of REST APIs. Modern access control methods such as OAuth2 allow APIs to decide what minimum priviledges they require for each endpoint, operation, or resource. In the same way, it gives them fine-grained control on what priviledges are assigned to API clients. However, for fine-grained control to be impactful, APIs need to carefully decide which priviledges they delegate to clients by default.

But why is it important that APIs assigned do not grant too many priviledges by default? The best practice for authorization is to operate on the so-called least priviledge principle. Basically, this means that a client or user should only get the minimum necessary priviledges required for the respective task they want to do. For default priviledges, the task is usually unspecified, so there are no necessary priviledges. In that case, we would expect an API to grant either no priviledges or the overall lowest functional priviledge level.

If the API uses OAuth2 as its access control method, we can easily test what the API considers default priviledges. In OAuth2, clients can request a specific level of priviledge via the scope parameter in the initial authorization request.

Including the scope parameter in the request is optional. If it's omitted, the API can deny the authorization request or - and that's what we are interested in - decide which scope it assigns to the authorization token returned to the client. By analyzing the default scope value, we can see whether the API adheres to the least priviledge principle.

REST-Attacker can automatically retrieve this information for configured OAuth2 clients with the scopes.TestTokenRequestScopeOmit test case. For every configured OAuth2 client, an authorization request without the scope parameter is sent to the OAuth2 authorzation endpoints of the API. The tool then extracts the scope that is assigned to the returned OAuth2 token. This scope value then has to be manually analyzed.

Out of the 6 APIs we tested, 2 (MS Graph and YouTube) denied requests without a scope parameter. The other 4 APIs (GitHub, Gitlab, Spotify, and Zoom) allowed omitting the scope parameter. Therefore, only the latter 4 APIs assigned default prviledges that could be analyzed.

API	Assigned Scope	Least Priviledge?
GitHub	`(none)`	Yes
Gitlab	`api`	No
Spotify	(default)	Yes*
Zoom	all approved	No

* OAuth2 scope with least priviledges

Interestingly, the extent to which a least priviledge principle was followed varied between APIs.

GitHub's API assigned the overall lowest possible priviledges by default via the (none) scope. With this scope, a client could only access API endpoints that were already publicly accessible (without providing authorization). While the scope does not grant more priviledges than a public client would get, the (none) scope had other benefits such as an increased rate limit.

In comparison, the Spotify API had no publicly accessible API endpoints and required authorization for every request. By default, tokens were assigned a "default" scope which was the OAuth2 scope with the lowest available priviledges and allowed clients to access several basic API endpoints.

Gitlab's and Zoom's API went into the opposite direction and assigned the highest priviledge to their clients by default. In Gitlab's case, this was the api scope which allowed read and write access to all API endpoints. Zoom required a pre-approval of scopes that the client wants to access during client registration. After registration, Zoom returned all approved scopes by default.

Conclusion

We've seen that while REST is not a clarly defined standard, this does not result in REST APIs being too complex for a generalized automated analysis. The usage of standardized HTTP components allows the design of simple yet effective tests that work across APIs. This also applies to other components that are used across APIs such as access control mechanisms like OAuth2. The practical tests we discussed worked on all APIs we tested, even if their underlying application contexts were different. However, we've also seen that most of the APIs were generally safe against these tests.

Tool-based automation could certainly play a much larger role in REST security, not only for finding security issues but also for filtering results and streamlining otherwise manual tasks. In the long run, this will hopefully also result in an increase in security.

Acknowledgement

The REST-Attacker project was developed as part of a master's thesis at the Chair of Network & Data Security of the Ruhr University Bochum. I would like to thank my supervisors Louis Jannett, Christian Mainka, Vladislav Mladenov, and Jörg Schwenk for their continued support during the development and review of the project.

More info

nefertiti

One year ago, we received a contract as a PDF file. It was digitally signed. We looked at the document - ignoring the "certificate is not trusted" warning shown by the viewer - and asked ourselfs:

"How do PDF signatures exactly work?"

We are quite familiar with the security of message formats like XML and JSON. But nobody had an idea, how PDFs really work. So we started our research journey.

Today, we are happy to announce our results. In this blog post, we give an overview how PDF signatures work and on top, we reveal three novel attack classes for spoofing a digitally signed PDF document. We present our evaluation of 22 different PDF viewers and show 21 of them to be vulnerable. We additionally evaluated 8 online validation services and found 6 to be vulnerable.

In cooperation with the BSI-CERT, we contacted all vendors, provided proof-of-concept exploits, and helped them to fix the issues and three generic CVEs for each attack class were issued: CVE-2018-16042, CVE-2018-18688, CVE-2018-18689.

Full results are available in the master thesis of Karsten Meyer zu Selhausen, in our security report, and on our website.

Digitally Signed PDFs? Who the Hell uses this?

Maybe you asked yourself, if signed PDFs are important and who uses them.
In fact, you may have already used them.
Have you ever opened an Invoice by companies such as Amazon, Sixt, or Decathlon?
These PDFs are digitally signed and protected against modifications.
In fact, PDF signatures are widely deployed in our world. In 2000, President Bill Clinton enacted a federal law facilitating the use of electronic and digital signatures in interstate and foreign commerce by ensuring the validity and legal effect of contracts. He approved the eSign Act by digitally signing it.
Since 2014, organizations delivering public digital services in an EU member state are required to support digitally signed documents, which are even admissible as evidence in legal proceedings.
In Austria, every governmental authority digitally signs any official document [§19]. In addition, any new law is legally valid after its announcement within a digitally signed PDF.
Several countries like Brazil, Canada, the Russian Federation, and Japan also use and accept digitally signed documents.
According to Adobe Sign, the company processed 8 billion electronic and digital signatures in the 2017 alone.

Crash Course: PDF and PDF Signatures

To understand how to spoof PDF Signatures, we unfortunately need to explain the basics first. So here is a breef overview.

PDF files are ASCII files. You can use a common text editor to open them and read the source code.

PDF header. The header is the first line within a PDF and defines the interpreter version to be used. The provided example uses version PDF 1.7.

PDF body. The body defines the content of the PDF and contains text blocks, fonts, images, and metadata regarding the file itself. The main building blocks within the body are objects. Each object starts with an object number followed by a generation number. The generation number should be incremented if additional changes are made to the object.

In the given example, the Body contains four objects: Catalog, Pages, Page, and stream. The Catalog object is the root object of the PDF file. It defines the document structure and can additionally declare access permissions. The Catalog refers to a Pages object which defines the number of the pages and a reference to each Page object (e.g., text columns). The Page object contains information how to build a single page. In the given example, it only contains a single string object "Hello World!".

Xref table. The Xref table contains information about the position (byte offset) of all PDF objects within the file.

Trailer. After a PDF file is read into memory, it is processed from the end to the beginning. By this means, the Trailer is the first processed content of a PDF file. It contains references to the Catalog and the Xref table.

How do PDF Signatures work?

PDF Signatures rely on a feature of the PDF specification called incremental saving (also known as incremental update), allowing the modification of a PDF file without changing the previous content.

As you can see in the figure on the left side, the original document is the same document as the one described above. By signing the document, an incremental saving is applied and the following content is added: a new Catalog, a Signature object, a new Xref table referencing the new object(s), and a new Trailer. The new Catalog extends the old one by adding a reference to the Signature object. The Signature object (5 0 obj) contains information regarding the applied cryptographic algorithms for hashing and signing the document. It additionally includes a Contents parameter containing a hex-encoded PKCS7 blob, which holds the certificates as well as the signature value created with the private key corresponding to the public key stored in the certificate. The ByteRange parameter defines which bytes of the PDF file are used as the hash input for the signature calculation and defines 2 integer tuples:
a, b : Beginning at byte offset a, the following b bytes are used as the first input for the hash calculation. Typically, a 0 is used to indicate that the beginning of the file is used while a b is the byte offset where the PKCS#7 blob begins.
c, d : Typically, byte offset c is the end of the PKCS#7 blob, while c d points to the last byte range of the PDF file and is used as the second input to the hash calculation.

According to the specification, it is recommended to sign the whole file except for the PKCS#7 blob (located in the range between a b and c).

Attacks

During our research, we discovered three novel attack classes on PDF signatures:

Universal Signature Forgery (USF)
Incremental Saving Attack (ISA)
Signature Wrapping Attack (SWA)

In this blog post, we give an overview on the attacks without going into technical details. If you are more interested, just take a look at the sources we summarized for you here.

Universal Signature Forgery (USF)

The main idea of Universal Signature Forgery (USF) is to manipulate the meta information in the signature in such a way that the targeted viewer application opens the PDF file, finds the signature, but is unable to find all necessary data for its validation.

Instead of treating the missing information as an error, it shows that the contained signature is valid. For example, the attacker can manipulate the Contents or ByteRange values within the Signature object. The manipulation of these entries is reasoned by the fact that we either remove the signature value or the information stating which content is signed.
The attack seems trivial, but even very good implementations like Adobe Reader DC preventing all other attacks were susceptible against USF.

Incremental Saving Attack (ISA)

The Incremental Saving Attack (ISA) abuses a legitimate feature of the PDF specification, which allows to update a PDF file by appending the changes. The feature is used, for example, to store PDF annotations, or to add new pages while editing the file.

The main idea of the ISA is to use the same technique for changing elements, such as texts, or whole pages included in the signed PDF file to what the attacker desires.
In other words, an attacker can redefine the document's structure and content using the Body Updates part. The digital signature within the PDF file protects precisely the part of the file defined in the ByteRange. Since the incremental saving appends the Body Updates to the end of the file, it is not part of the defined ByteRange and thus not part of the signature's integrity protection. Summarized, the signature remains valid, while the Body Updates changed the displayed content.
This is not forbidden by the PDF specification, but the signature validation should indicate that the document has been altered after signing.

Signature Wrapping Attack (SWA)

Independently of the PDFs, the main idea behind Signature Wrapping Attacks is to force the verification logic to process different data than the application logic.

In PDF files, SWA targets the signature validation logic by relocating the originally signed content to a different position within the document and inserting new content at the allocated position. The starting point for the attack is the manipulation of the ByteRange value allowing to shift the signed content to different loctions within the file.

On a very technical level, the attacker uses a validly signed document (shown on the left side) and proceeds as follows:

Step 1 (optional): The attacker deletes the padded zero Bytes within the Contents parameter to increase the available space for injecting manipulated objects.
Step 2: The attacker defines a new /ByteRange [a b c* d] by manipulating the c value, which now points to the second signed part placed on a different position within the document.
Step 3: The attacker creates a new Xref table pointing to the new objects. It is essential that the byte offset of the newly inserted Xref table has the same byte offset as the previous Xref table. The position is not changeable since it is refer- enced by the signed Trailer. For this purpose, the attacker can add a padding block (e.g., using whitespaces) before the new Xref table to fill the unused space.
Step 4: The attacker injects malicious objects which are not protected by the signature. There are different injection points for these objects. They can be placed before or after the malicious Xref table. If Step 1 is not executed, it is only possible to place them after the malicious Xref table.
Step 5 (optional): Some PDF viewers need a Trailer after the manipulated Xref table, otherwise they cannot open the PDF file or detect the manipulation and display a warning message. Copying the last Trailer is sufficient to bypass this limitation.
Step 6: The attacker moves the signed content defined by c and d at byte offset c*. Optionally, the moved content can be encapsulated within a stream object. Noteworthy is the fact that the manipulated PDF file does not end with %%EOF after the endstream. The reason why some validators throw a warning that the file was manipulated after signing is because of an %%EOF after the signed one. To bypass this requirement, the PDF file is not correctly closed. However, it will be still processed by any viewer.

Evaluation

In our evaluation, we searched for desktop applications validating digitally signed PDF files. We analyzed the security of their signature validation process against our 3 attack classes. The 22 applications fulfill these requirements. We evaluated the latest versions of the applications on all supported platforms (Windows, MacOS, and Linux).

Authors of this Post

Vladislav Mladenov
Christian Mainka
Karsten Meyer zu Selhausen
Martin Grothe
Jörg Schwenk

Acknowledgements

Many thanks to the CERT-Bund team for the great support during the responsible disclosure.
We also want to acknowledge the teams which reacted to our report and fixed the vulnerable implementations.