1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
|
# Test Moderator CLI
A command-line tool for testing the intelligent in-character moderator against text files before deploying it to production.
## Features
- **File Testing**: Test individual text files or entire directories
- **Mock Submissions**: Creates realistic mock Reddit submissions for testing
- **AI Evaluation**: Uses the same GPT-5-nano evaluation logic as the production bot
- **Detailed Results**: Shows AI reasoning, confidence scores, and proposed actions
- **Mod Mail Preview**: Shows what mod mail messages would be sent to users
## Usage
### Prerequisites
1. **OpenAI API Key**: You need a valid OpenAI API key with access to GPT-5-nano
2. **Python Dependencies**: Install the required packages:
```bash
pip install openai
```
### Basic Usage
```bash
# Test a single file
python test_moderator.py --file sample_post.txt
# Test all .txt files in a directory
python test_moderator.py --directory test_posts/
# Test with custom author name
python test_moderator.py --file post.txt --author myuser
# Use API key from command line
python test_moderator.py --file post.txt --api-key sk-your-key-here
```
### Environment Variables
The tool automatically loads configuration from a `.env` file in the project root. Create a `.env` file with:
```bash
# OpenAI API Credentials
OPENAI_API_KEY=sk-your-key-here
```
Alternatively, you can set environment variables directly:
```bash
export OPENAI_API_KEY="sk-your-key-here"
```
### Command Line Options
- `--file, -f`: Test a single text file
- `--directory, -d`: Test all .txt files in a directory
- `--author, -a`: Author name for mock submissions (default: testuser)
- `--api-key, -k`: OpenAI API key (or set OPENAI_API_KEY env var)
- `--verbose, -v`: Show detailed output
## Sample Test Files
The `test_posts/` directory contains sample files demonstrating different types of content:
- **`artwork_showcase.txt`**: Art-focused post that should be re-flaired
- **`high_quality_roleplay.txt`**: Well-written in-character content that should be allowed
- **`low_effort_post.txt`**: Minimal content that should be removed
- **`basic_roleplay.txt`**: Simple in-character content that might be borderline
- **`poetic_roleplay.txt`**: Creative, high-quality in-character content
## Understanding Results
### AI Evaluation Output
The tool shows:
- **Should be Art**: Whether the post should be re-flaired as Art
- **Is Low Effort**: Whether the post should be removed for low-effort
- **Confidence**: AI confidence score (0.0 to 1.0)
- **Reasoning**: Detailed explanation of the AI's decision
### Actions
- **🎨 Change flair to Art**: Post will be re-flaired and user notified
- **🗑️ Remove post (low-effort)**: Post will be removed and user notified
- **✅ Allow post**: Post passes all checks and will be allowed
### Mod Mail Preview
The tool shows exactly what mod mail messages would be sent to users, including:
- Subject lines
- Message content
- Reasoning explanations
## Example Output
```
1. test_posts/artwork_showcase.txt
------------------------------------------------------------
Title: Artwork Showcase
Word Count: 45
Has Media: True
Media Type: text
Content Preview: Check out this amazing artwork I drew of Tokai Teio! She's so beautiful and I spent hours getting every detail perfect...
🤖 AI Evaluation:
Should be Art: True
Is Low Effort: False
Confidence: 0.95
Reasoning: This post is primarily showcasing artwork with detailed description of the art process, making it more suitable for the Art flair than In Character.
📋 Actions:
🎨 Change flair to Art
📧 Mod Mail (Art Flair Change):
Subject: Your post flair has been changed to Art
Message: Your in-character post has been automatically re-flaired as 'Art' because it appears to be primarily showcasing artwork or visual content rather than in-character content.
Reasoning: This post is primarily showcasing artwork with detailed description of the art process, making it more suitable for the Art flair than In Character.
```
## Setup
1. **Install Dependencies**: `pip install openai python-dotenv`
2. **Create .env File**: Add `OPENAI_API_KEY=sk-your-key-here` to your `.env` file
3. **Run Tests**: `python test_moderator.py --directory test_posts/`
## Tips for Testing
1. **Test Edge Cases**: Include borderline content to see how the AI handles ambiguous cases
2. **Check Confidence Scores**: Low confidence scores indicate the AI is uncertain
3. **Review Reasoning**: The AI's explanations help understand its decision-making process
4. **Test Different Lengths**: Try very short posts, very long posts, and everything in between
5. **Include Media References**: Test posts that mention images, videos, or other media
## Troubleshooting
### Common Issues
- **API Key Error**: Make sure your OpenAI API key is valid and has access to GPT-5-nano
- **File Not Found**: Check that file paths are correct and files exist
- **JSON Parse Error**: The AI sometimes returns malformed JSON; the tool handles this gracefully
- **Rate Limiting**: If you hit OpenAI rate limits, wait a moment and try again
### Getting Help
If you encounter issues:
1. Check that all dependencies are installed
2. Verify your OpenAI API key is working
3. Ensure test files are valid text files
4. Check the error messages for specific guidance
|