Prompt Engineering for Generative AI: Future-Proof Inputs for Reliable AI Outputs at Scale
James Phoenix and Mike Taylor
Table of Contents
Preface. . . . . . . . . . . . . . . . . . . . . .. . . . . . . xi
- The Five Principles of Prompting. . . . . . . . . . . . .. . . . . 1
Overview of the Five Principles of Prompting 4
- Give Direction 8
- Specify Format 14
- Provide Examples 17
- Evaluate Quality 20
- Divide Labor 32
Summary 38
- Introduction to Large Language Models for Text Generation. . . . . .. 41
What Are Text Generation Models? 41
Vector Representations: The Numerical Essence of Language 42
Transformer Architecture: Orchestrating Contextual Relationships 43
Probabilistic Text Generation: The Decision Mechanism 45
Historical Underpinnings: The Rise of Transformer Architectures 46
OpenAI’s Generative Pretrained Transformers 48
GPT-3.5-turbo and ChatGPT 48
GPT-4 51
Google’s Gemini 51
Meta’s Llama and Open Source 52
Leveraging Quantization and LoRA 53
Mistral 53
Anthropic: Claude 54
GPT-4V(ision) 54
Model Comparison 54
Summary 55
- Standard Practices for Text Generation with ChatGPT. . . . .. . . 57
Generating Lists 57
Hierarchical List Generation 59
When to Avoid Using Regular Expressions 64
Generating JSON 64
YAML 66
Filtering YAML Payloads 67
Handling Invalid Payloads in YAML 69
Diverse Format Generation with ChatGPT 72
Mock CSV Data 73
Explain It like I’m Five 74
Universal Translation Through LLMs 74
Ask for Context 76
Text Style Unbundling 80
Identifying the Desired Textual Features 80
Generating New Content with the Extracted Features 82
Extracting Specific Textual Features with LLMs 82
Summarization 82
Summarizing Given Context Window Limitations 83
Chunking Text 85
Benefits of Chunking Text 86
Scenarios for Chunking Text 86
Poor Chunking Example 87
Chunking Strategies 89
Sentence Detection Using SpaCy 90
Building a Simple Chunking Algorithm in Python 91
Sliding Window Chunking 92
Text Chunking Packages 94
Text Chunking with Tiktoken 95
Encodings 95
Understanding the Tokenization of Strings 95
Estimating Token Usage for Chat API Calls 97
Sentiment Analysis 99
Techniques for Improving Sentiment Analysis 101
Limitations and Challenges in Sentiment Analysis 101
Least to Most 102
Planning the Architecture 102
Coding Individual Functions 102
Adding Tests 103
Benefits of the Least to Most Technique 104
Challenges with the Least to Most Technique 105
Role Prompting 105
Benefits of Role Prompting 106
Challenges of Role Prompting 106
When to Use Role Prompting 107
GPT Prompting Tactics 107
Avoiding Hallucinations with Reference 108
Give GPTs “Thinking Time” 110
The Inner Monologue Tactic 111
Self-Eval LLM Responses 112
Classification with LLMs 114
Building a Classification Model 115
Majority Vote for Classification 116
Criteria Evaluation 117
Meta Prompting 120
Summary 124
- Advanced Techniques for Text Generation with LangChain. . . .. 125
Introduction to LangChain 125
Environment Setup 127
Chat Models 128
Streaming Chat Models 130
Creating Multiple LLM Generations 130
LangChain Prompt Templates 131
LangChain Expression Language (LCEL) 132
Using PromptTemplate with Chat Models 134
Output Parsers 134
LangChain Evals 138
OpenAI Function Calling 146
Parallel Function Calling 149
Function Calling in LangChain 151
Extracting Data with LangChain 152
Query Planning 153
Creating Few-Shot Prompt Templates 154
Fixed-Length Few-Shot Examples 155
Formatting the Examples 155
Selecting Few-Shot Examples by Length 157
Limitations with Few-Shot Examples 159
Saving and Loading LLM Prompts 159
Data Connection 161
Document Loaders 162
Text Splitters 165
Text Splitting by Length and Token Size 165
Text Splitting with Recursive Character Splitting 167
Task Decomposition 169
Prompt Chaining 171
Sequential Chain 172
itemgetter and Dictionary Key Extraction 173
Structuring LCEL Chains 178
Document Chains 179
Stuff 181
Refine 182
Map Reduce 182
Map Re-rank 183
Summary 184
- Vector Databases with FAISS and Pinecone. . . .. . . 185
Retrieval Augmented Generation (RAG) 188
Introducing Embeddings 189
Document Loading 198
Memory Retrieval with FAISS 201
RAG with LangChain 206
Hosted Vector Databases with Pinecone 207
Self-Querying 215
Alternative Retrieval Mechanisms 220
Summary 221
- Autonomous Agents with Memory and Tools. . . . .. 223
Chain-of-Thought 223
Agents 225
Reason and Act (ReAct) 227
Reason and Act Implementation 229
Using Tools 235
Using LLMs as an API (OpenAI Functions) 237
Comparing OpenAI Functions and ReAct 241
Use Cases for OpenAI Functions 242
ReAct 242
Use Cases for ReAct 243
Agent Toolkits 243
Customizing Standard Agents 245
Custom Agents in LCEL 246
Understanding and Using Memory 249
Long-Term Memory 249
Short-Term Memory 249
Short-Term Memory in QA Conversation Agents 250
Memory in LangChain 250
Preserving the State 251
Querying the State 252
ConversationBufferMemory 252
Other Popular Memory Types in LangChain 255
ConversationBufferWindowMemory 255
ConversationSummaryMemory 255
ConversationSummaryBufferMemory 256
ConversationTokenBufferMemory 256
OpenAI Functions Agent with Memory 257
Advanced Agent Frameworks 259
Plan-and-Execute Agents 259
Tree of Thoughts 260
Callbacks 261
Global (Constructor) Callbacks 263
Request-Specific Callbacks 264
The Verbose Argument 264
When to Use Which? 264
Token Counting with LangChain 264
Summary 266
- Introduction to Diffusion Models for Image Generation. . . . . 267
OpenAI DALL-E 271
Midjourney 273
Stable Diffusion 277
Google Gemini 279
Text to Video 279
Model Comparison 279
Summary 280
- Standard Practices for Image Generation with Midjourney. 283
Format Modifiers 283
Art Style Modifiers 287
Reverse Engineering Prompts 289
Quality Boosters 290
Negative Prompts 292
Weighted Terms 294
Prompting with an Image 297
Inpainting 300
Outpainting 303
Consistent Characters 305
Prompt Rewriting 307
Meme Unbundling 310
Meme Mapping 315
Prompt Analysis 317
Summary 318
- Advanced Techniques for Image Generation with Stable Diffusion. . . . 319
Running Stable Diffusion 319
AUTOMATIC1111 Web User Interface 326
Img2Img 334
Upscaling Images 337
Interrogate CLIP 339
SD Inpainting and Outpainting 340
ControlNet 343
Segment Anything Model (SAM) 352
DreamBooth Fine-Tuning 355
Stable Diffusion XL Refiner 362
Summary 365
- Building AI-Powered Applications. . . . . . .. . 367
AI Blog Writing 367
Topic Research 368
Expert Interview 371
Generate Outline 373
Text Generation 374
Writing Style 377
Title Optimization 380
AI Blog Images 381
User Interface 387
Summary 389
Index. . . . . . . . . . . . . . . . . . . . . .. . . . . 391