What’s an OpenAI Assistant? Think of it as a software glue that affords you to gel together agent-like capabilities in your applications to conduct tasks expressed as instructions in natural language to an Assistant. Able to understand instructions, it can leverage OpenAI’s SOTA models and tools to carry out tasks. With Assistants stateful API, you can create Assistants within your application, providing you access to three types of supported tools: Code Interpreter, Retrieval, and Function calling [5]. At the core it has few concepts and components that cogently interact together, to enable agent-like capabilities.

Assistants API, concepts, components, and tools Unfortunately, OpenAI documentation falls short in explaining or illustrating these components into finer details and showing how they work together. Randy Michak of Empowerment AI does a fine job of dissecting these core components and illustrating their flow and data interactions [7]. Inspired by Michak, I mildly modified Figure 4, showing dynamic interaction and data flowing among Assistants API components.

To get started with Assistants, the OpenAI guide stipulates four simple steps to use the Assistants API to glue together these core components for coordination [8]. Step 1: Create an Assistant, to declare a custom model and provide instructions for the Assistant. This helps the Assistant to elect the appropriate supported tool to employ. Step 2: Create a Thread, a stateful session for the Assistant to retrieve messages from and add Assistant messages to. Step 3: Use the Thread as a conversational session to add messages for the assistants to consume. Step 4: Run the Assistant on a newly added Thread message to trigger responses. This run is Assistant’s asynchronous runtime environment.

How does it all work together?

Let’s methodically walk through a simple example where we want to accomplish the following: Integrate Assistants API, using Retrieval tool, to a) upload a couple of pdf documents and b) use an Assistant to query the contents of the document. Consider this as a mini Retrieval Augmented Generation (RAG) application. Use Files objects to upload the pdf files so that the Assistant can access them. Create and employ the Assistant, Threads, Messages, and Run objects to query the uploaded pdf documents. Coordinate all these concrete objects to interact and interplay together as part of my application.

Step 1: Create File objects as our knowledge base Upload your PDFs in the retrievers’ database, using a File object. The Assistants API breaks them into parts, as chunks, and saves them, as indexes and vector embeddings. When you ask a question, Retrievers find the best matches and help the Assistant give you a detailed answer, just like a big RAG retriever.

Step 2: Create an Assistant object. To use an Assistant and conduct tasks, first, create an AI Assistant object. Supply the Assistant with a model, instructional behavior, tools to use, and file IDs to employ for its knowledge base, as parameters.

Both OpenAI programing guide and Anyscale Endpoints blog [7] distill down to simple steps: Call the model with the user query and a list of functions defined in the Chat Completions API parameter as tools. The model can choose to call one or more functions; if so, the content will be a stringified JSON object adhering to your custom schema. Parse the string into JSON in your code, and call your function with the provided arguments if they exist. Call the model again by appending the function response as a new message, and let the model summarize the results back to the user. Following the above simple steps, our user_content to the LLM generates three required parameters (location, latitude, longitude) as a JSON object in its response.

Examples and Use Cases of Function Calling in LLM

Apart from the above use cases mentioned in the Open AI programming guide [10], Ben Lorica visually and comprehensively captures use cases of general function calling in LLMs, including the OpenAI Assistant Tools API [11]. Lorica succinctly states that early use cases include applications such as customer service chatbots, data analysis assistants, and code generation tools. Other examples extend to creative, logistical, and operational domains: writing assistants, scheduling agents, summarizing news., etc.

From the beginning, Remix's opinion has always been that you own your server architecture. This is why Remix is built on top of the Web Fetch API and can run on any modern runtime via built-in or community-provided adapters. While we believe that having a server provides the best UX/Performance/SEO/etc. for most apps, it is also undeniable that there exist plenty of valid use cases for a Single Page Application in the real world:

SPA Mode is basically what you'd get if you had your own React Router + Vite setup using createBrowserRouter/RouterProvider, but along with some extra Remix goodies: File-based routing (or config-based via routes()) Automatic route-based code-splitting via route.lazy <Link prefetch> support to eagerly prefetch route modules <head> management via Remix <Meta>/<Links> APIs SPA Mode tells Remix that you do not plan on running a Remix server at runtime and that you wish to generate a static index.html file at build time and you will only use Client Data APIs for data loading and mutations. The index.html is generated from the HydrateFallback component in your root.tsx route. The initial "render" to generate the index.html will not include any routes deeper than root. This ensures that the index.html file can be served/hydrated for paths beyond / (i.e., /about) if you configure your CDN/server to do so.

inurl:”/api/v1" site:microsoft.com

Many times when the we are trying to Pentest an API we might not get access to Swagger file or the documentations of the API, Today we will try to create the swagger file using Mitmweb and Postman.

Man in The Midlle Proxy (MITMweb)

run mitmweb through our command line in Kali

and as we can see it starts to listen on the port 8080 for http/https traffic, and we will make sure that its running by navigating to the above address which is the localhost at port 8081

and then we will proxy our traffic thorugh Burp Suite proxy port 8080 because we already has mitmweb listening for this port (make sure Burp is closed)

and then we will stop the capture and use mitmproxy2swagger to analyse it

When the Browser is the client consuming HTML, it understands how to render HTML. HTML has a specification. The browser understands how to handle a <form> tag or a <button>. It was driven by the HTML at runtime.

How can you build a smarter Single Page Application with a REST API? The concepts have been since the beginning of the web, yet have somehow lost their way in modern REST API that drives a Single Page Application or Mobile Applications. Here’s how to guide clients based on state by moving more information from design time to runtime.

State If you’re developing more than a CRUD application, you’re likely going to be driven by the state of the system. Apps that have Task Based UIs (hint: go read my post on Decomposing CRUD to Task Based UIs) are guiding users down a path of actions they can perform based on the state of the system. The example throughout this post is the concept of a Product in a warehouse. If we have a tasks that let’s someone mark a Product as no longer being available for sale or it being available for sale, these tasks can be driven by the state of the Product. If the given UI task is “Mark as Available” then the Product must be currently unavailable and we have a quantity on hand that’s greater than zero

History of Clients Taking a step back a bit, web apps were developed initially with just plain HTML (over 20 years ago for me). In its most basic form, a static HTML page contained a <form> that the browser rendered for the user to fill out and submit. The form’s action would point to a URI usually to a script, often written in Perl, in the cgi-bin folder on the webserver. The script would take the form data (sent via POST from the browser) and insert it into a database, send an email, or whatever the required behavior was. As web apps progressed, instead of the HTML being in a static file, it was dynamically created by the server. But it was still just plain HTML. The browser was the client. HTML was the content it’s consuming.

Modern Clients As web apps progressed with AJAX (XMLHttpRequest) instead of using HTML forms, Javascript was used to send the HTTP request. The browsers turned more into the Host of the application which was written in Javascript. Now, Javascript is the client. JSON is the content it’s consuming.

Runtime vs Design Time When the Browser is the client consuming HTML, it understands how to render HTML. HTML has a specification. The browser understands how to handle a <form> tag or a <button>. It was driven by the HTML at runtime.

In modern SPAs consuming JSON, the data itself is unstructured. Each client has to be created uniquely based on the content it receives. This has to be developed at design time when creating the javascript client.

When developing a SPA, you may leverage something like OpenAPI to generate code to use in the SPA/clients to make the HTTP calls to the server. But you must understand as a developer, at design time (when developing) when to make a call to the server.

To use my earlier example of making a product available for sale, if you were developing a server-side rendered HTML web app, you wouldn’t return the form apart of the HTML if the product couldn’t be made available. You would do this because on the server you have the state of the product (fetched from the database). If you’re creating a SPA, you’re likely putting that same logic in your client so you can conditionally show UI elements. It wouldn’t be a great experience for the user to be able to perform an action, then see an error message because the server/api threw a 400 because the product is not in a state to allow it to be available.

Hypermedia Hypermedia is what is used in HTML to tell the Browser what it can do. As I mentioned earlier, a <form> is a hypermedia control.

HTTP APIs The vast majority of modern HTTP APIs serving JSON, do not provide any information in the content (JSON) about what actions or other resources the consuming client (SPA) can take. Meaning, we provide no information at runtime. All of that has to be figured out at design time.

You will still need to know (via OpenAPI) at design time, all the information about the routes you will be calling, and their results, however, you can now have the server return JSON that can guide the client based on state.