9553205131

Committed 17 Jun 2024 04:41PM UTC coverage: 32.002% (+0.2%) from 31.836%

Build # 9553205131

Build Type

Pull #9301

github

Committed by

oetherington

Commit Message

Merge branch 'master' into normalized-posts-contents

Pull Request Pull Request #9301: Normalized posts contents

Run Details

4909 of 39600 branches covered (12.4%)

Branch coverage included in aggregate %.

125 of 234 new or added lines in 43 files covered. (53.42%)

11 existing lines in 10 files now uncovered.

28204 of 63871 relevant lines covered (44.16%)

190.99 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

15.28

/packages/lesswrong/server/scripts/generativeModels/coverImage.ts

import OpenAI from 'openai';
import { Globals, createAdminContext, createMutator } from '../../vulcan-lib/index.ts';
import ReviewWinners from '../../../lib/collections/reviewWinners/collection.ts';
import ReviewWinnerArts from '../../../lib/collections/reviewWinnerArts/collection.ts';
import { moveImageToCloudinary } from '../convertImagesToCloudinary.ts';
import { myMidjourneyAPIKeySetting } from '../../../lib/instanceSettings.ts';
import { getOpenAI } from '../../languageModels/languageModelIntegration.ts';
import { sleep } from '../../../lib/utils/asyncUtils.ts';
import shuffle from 'lodash/shuffle';
import { filterNonnull } from '../../../lib/utils/typeGuardUtils.ts';
import { fetchFragment } from '../../fetchFragment.ts';

const myMidjourneyKey = myMidjourneyAPIKeySetting.get()

const promptUrls = [
  "https://s.mj.run/W91s58GkTUs",
  "https://s.mj.run/D5okH4Ak-mw",
  "https://s.mj.run/1aM-y0W73aA",
  "https://s.mj.run/JAlgYmUiOdc",
]

const llm_prompt = (title: string, essay: string) => `I am creating cover art for essays that will be featured on LessWrong. For each piece of art, I want a clear visual metaphor that captures the essence of the essay.

The visual metaphor should be concrete and specific, and should be something that can be depicted in a single image. The metaphor should be something that is both visually striking and that captures the essence of the essay in a way that is likely to be interesting. It should be 5 - 15 words long.

The image should not contain any text. It should not have any writing. It should not refer to the content of written materials. It should not ask for symbols representing concepts, but instead ask for concrete images (it's fine if you intend them to represent something, but they should be concrete images).

If the essay contains any particular images or visual metaphors, feel free to use those in the answers.

Here are some examples:

1. a sea of thousands of people, one of them zoomed in using a magnifying glass
2. a set of scales with a heap of gold on one side and a heart on the other
3. a tree with tangled roots and a single leaf
4. a person standing on a mountain peak looking out over a vast landscape
5. images from different time periods with a wanderer walking through them

Here are some bad examples:

1. A quill writing the word 'honor'
2. A pile of resources dwindling
3. A collection of books about Zen Buddhism

Please generate 3 visual metaphors for the essay that will appear on Lesswrong. The essay will appear after the "===".

Please format your response as follows:

SUMMARY: What is the main idea of the essay?
IDEAS: A JSON list of 3 visual metaphors, like ["a sea of thousands of people, one of them zoomed in using a magnifying glass", "a set of scales with a heap of gold on one side and a heart on the other", "a tree with tangled roots and a single leaf"]
CHECK: For each metaphor, write out the metaphor and answer (a) does the metaphor contain writing or refer to the content of written materials or say that words should appear in the image? (yes/no) (b) Does the metaphor ask for any symbols respresenting concepts? (yes/no) Is it 5 to 15 words long? (yes/no)
CORRECTIONS: If any of the metaphors contain writing or refer to the content of written materials, please provide a corrected version of the metaphor.
METAPHORS: A JSON list of your final 3 visual metaphors.

===

${title}

${essay}`

const getEssays = async (): Promise<Essay[]> => {
  const postIds = await ReviewWinners
    .find({}, { projection: { postId: 1 }, sort: { reviewRanking: 1 } })
    .fetch();
  const reviewArts = await ReviewWinnerArts
    .find({})
    .fetch();
  const postIdsWithoutLotsOfArt = postIds
  .filter(p => reviewArts.filter(a => a.postId === p.postId).length < 9)
  const postIdsWithoutEnoughArt = postIds
  .filter(p => reviewArts.filter(a => a.postId === p.postId).length < 10)

  const postsToFind = postIdsWithoutLotsOfArt.length > 0 ? postIdsWithoutLotsOfArt : postIdsWithoutEnoughArt

  const essays = await fetchFragment({
    collectionName: "Posts",
    fragmentName: "PostsPage",
    selector: {_id: {$in: postsToFind.map(p => p.postId)}},
    currentUser: null,
    skipFiltering: true,
  });

  const toGenerate = (p: PostsPage) => Math.ceil((12 - reviewArts.filter(art => art.postId === p._id).length)/4)

  return essays.map(e => {
    return {post: e, title: e.title, content: e.contents?.html ?? "", toGenerate: toGenerate(e) }
  })
}

type Essay = {post: PostsPage, title: string, content: string, toGenerate: number}
type MyMidjourneyResponse = {messageId: "string", uri?: string, progress: number, error?: string}

const getElements = async (openAiClient: OpenAI, essay: {title: string, content: string}, tryCount = 0): Promise<string[]> => {
  const content = essay.content.length > 25_000 ? essay.content.slice(0, 12_500) + "\n[EXCERPTED FOR LENGTH]\n" + essay.content.slice(-12_500) : essay.content
  const completion = await openAiClient.chat.completions.create({
    messages: [{role: "user", content: llm_prompt(essay.title, content)}],
    model: "gpt-4",
  }).catch((error) => {
    if (error instanceof OpenAI.APIError && error.status === 400 && error.code === 'context_length_exceeded') {
      return openAiClient.chat.completions.create({
        messages: [{role: "user", content: llm_prompt(essay.title, content.slice(0, 8_000) + "\n[EXCERPTED FOR LENGTH]\n" + essay.content.slice(-8_000))}],
        model: "gpt-4",
      })
    } else {
      // eslint-disable-next-line no-console
      console.error(error)
      return undefined
    }
  })


  try {
    return JSON.parse((completion?.choices[0].message.content || '').split('METAPHORS: ')[1])
  } catch (error) {
    // eslint-disable-next-line no-console
    console.error('Error parsing response:', error);
    if (tryCount < 2) return getElements(openAiClient, essay, tryCount + 1)
    return []
  }
}

const prompter = (el: string) => {
  const lowerCased = el[0].toLowerCase() + el.slice(1)
  return `${shuffle(promptUrls)[0]} topographic watercolor artwork of ${lowerCased}, in the style of ethereal watercolor washes, ultrafine detail, juxtaposition of hard and soft lines, delicate ink lines, inspired by scientific illustrations, in the style of meditative pastel moebius, muted colors --ar 8:5 --v 6.0 `
}

const pressMidjourneyButton = async (messageId: string, button: string) => {
  return fetch(`https://api.mymidjourney.ai/api/v1/midjourney/button`, {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${myMidjourneyKey}`,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({messageId, button})
  }).then(r => r.json())
    .then(j => j.messageId)
    .then(checkOnJob)
}

const saveImage = async (el: string, essay: Essay, url: string) => {
  const newUrl = await moveImageToCloudinary(url, `splashArtImagePrompt${el}`)
  if (!newUrl) {
    // eslint-disable-next-line no-console
    console.error("Failed to upload image to cloudinary", el, essay)
    return
  }
  await createMutator({
    collection: ReviewWinnerArts,
    context: createAdminContext(),
    document: {
      postId: essay.post._id, 
      splashArtImagePrompt: el,
      splashArtImageUrl: newUrl
    }
  })
  return newUrl
}

const upscaledImages = async (el: string, essay: Essay, messageId: string): Promise<(string | undefined)[]> =>
  Promise.all(["U1","U2","U3","U4"]
    .map(async button => {
      return pressMidjourneyButton(messageId, button)
        .then(m => m && upscaleImage(m.messageId))
        .then(uri => uri && saveImage(el, essay, uri))
    }))

const upscaleImage = async (messageId: string): Promise<string | undefined> => {
  const res = await pressMidjourneyButton(messageId, "Upscale (Subtle)")
  return res?.uri
}

async function checkOnJob(jobId: string): Promise<MyMidjourneyResponse | undefined> {
  try {
    const response = await fetch(`https://api.mymidjourney.ai/api/v1/midjourney/message/${jobId}`, {
      method: 'GET',
      headers: {
        'Authorization': `Bearer ${myMidjourneyKey}`,
        'Content-Type': 'application/json'
      }
    })

    const responseData = await response.json()
    if (responseData.progress === 100 || responseData.error) {
      if (responseData.error) {
        // eslint-disable-next-line no-console
        console.error('Image generation failed:', responseData);
        return undefined
      }
      return responseData
    } else {
      await sleep(5_000)
      return checkOnJob(jobId)
    }
  } catch (error) {
    // eslint-disable-next-line no-console
    console.error('Error getting updates', error);
    return checkOnJob(jobId)
  }
}

async function getEssayPromptJointImageMessage(promptElement: string): Promise<MyMidjourneyResponse|undefined> {
  let promptResponseData: any;
  let jobId: string;  
  try {
    const response = await fetch('https://api.mymidjourney.ai/api/v1/midjourney/imagine', {
      method: 'POST',
      headers: {
        'Authorization': `Bearer ${myMidjourneyKey}`,
        'Content-Type': 'application/json'
      },
      body: JSON.stringify({prompt: prompter(promptElement)})
    });

    promptResponseData = await response.json();
    jobId = promptResponseData.messageId
    return checkOnJob(jobId)
  } catch (error) {
    // eslint-disable-next-line no-console
    console.error('Error generating image:', error);
  }

}

async function generateCoverImages({limit = 2}: {
    limit?: number
} = {}): Promise<string[]> {
  const openAiClient = await getOpenAI()
  if (!openAiClient) {
    throw new Error('Could not initialize OpenAI client!');
  }
  const essays = (await getEssays()).slice(0, limit)

  return await essays.reduce(async (prev: Promise<string[]>, essay: Essay): Promise<string[]> => {

    const prevUrls = await prev
    const images = await getElements(openAiClient, essay)
      .then((els: string[]) =>
        els.slice(0,Math.min(limit, essay.toGenerate))
          .reduce(async (prev, el) => {
            const ims = await prev
            const im = await getEssayPromptJointImageMessage(el)
              .then(x => x === undefined ? Promise.resolve([]) : upscaledImages(el, essay, x.messageId))
              .then(urls => filterNonnull(urls))
            return [...ims, ...im]
          }, Promise.resolve([]) as Promise<string[]>))

    return [...prevUrls, ...images]
  }, Promise.resolve([]) as Promise<string[]>)
}

async function coverImages () {
  if (!myMidjourneyKey) {
    throw new Error('No MyMidjourney API key found!');
  }
  
  const newImages = await generateCoverImages({limit: 9999})
  if (newImages.length === 0) return
  await sleep(10_000)
  // There are often failures in the process of the generation. We run until we no longer have candidate essays
  await coverImages()
}

Globals.coverImages = coverImages

1	import OpenAI from 'openai';	18✔
2	import { Globals, createAdminContext, createMutator } from '../../vulcan-lib/index.ts';	18✔
3	import ReviewWinners from '../../../lib/collections/reviewWinners/collection.ts';	18✔
4	import ReviewWinnerArts from '../../../lib/collections/reviewWinnerArts/collection.ts';	18✔
5	import { moveImageToCloudinary } from '../convertImagesToCloudinary.ts';	18✔
6	import { myMidjourneyAPIKeySetting } from '../../../lib/instanceSettings.ts';	18✔
7	import { getOpenAI } from '../../languageModels/languageModelIntegration.ts';	18✔
8	import { sleep } from '../../../lib/utils/asyncUtils.ts';	18✔
9	import shuffle from 'lodash/shuffle';	18✔
10	import { filterNonnull } from '../../../lib/utils/typeGuardUtils.ts';	18✔
11	import { fetchFragment } from '../../fetchFragment.ts';	18✔
12
13	const myMidjourneyKey = myMidjourneyAPIKeySetting.get()	18✔
14
15	const promptUrls = [	18✔
16	"https://s.mj.run/W91s58GkTUs",
17	"https://s.mj.run/D5okH4Ak-mw",
18	"https://s.mj.run/1aM-y0W73aA",
19	"https://s.mj.run/JAlgYmUiOdc",
20	]
21
22	const llm_prompt = (title: string, essay: string) => `I am creating cover art for essays that will be featured on LessWrong. For each piece of art, I want a clear visual metaphor that captures the essence of the essay.	18✔
23
24	The visual metaphor should be concrete and specific, and should be something that can be depicted in a single image. The metaphor should be something that is both visually striking and that captures the essence of the essay in a way that is likely to be interesting. It should be 5 - 15 words long.
25
26	The image should not contain any text. It should not have any writing. It should not refer to the content of written materials. It should not ask for symbols representing concepts, but instead ask for concrete images (it's fine if you intend them to represent something, but they should be concrete images).
27
28	If the essay contains any particular images or visual metaphors, feel free to use those in the answers.
29
30	Here are some examples:
31
32	1. a sea of thousands of people, one of them zoomed in using a magnifying glass
33	2. a set of scales with a heap of gold on one side and a heart on the other
34	3. a tree with tangled roots and a single leaf
35	4. a person standing on a mountain peak looking out over a vast landscape
36	5. images from different time periods with a wanderer walking through them
37
38	Here are some bad examples:
39
40	1. A quill writing the word 'honor'
41	2. A pile of resources dwindling
42	3. A collection of books about Zen Buddhism
43
44	Please generate 3 visual metaphors for the essay that will appear on Lesswrong. The essay will appear after the "===".
45
46	Please format your response as follows:
47
48	SUMMARY: What is the main idea of the essay?
49	IDEAS: A JSON list of 3 visual metaphors, like ["a sea of thousands of people, one of them zoomed in using a magnifying glass", "a set of scales with a heap of gold on one side and a heart on the other", "a tree with tangled roots and a single leaf"]
50	CHECK: For each metaphor, write out the metaphor and answer (a) does the metaphor contain writing or refer to the content of written materials or say that words should appear in the image? (yes/no) (b) Does the metaphor ask for any symbols respresenting concepts? (yes/no) Is it 5 to 15 words long? (yes/no)
51	CORRECTIONS: If any of the metaphors contain writing or refer to the content of written materials, please provide a corrected version of the metaphor.
52	METAPHORS: A JSON list of your final 3 visual metaphors.
53
54	===
55
56	${title}
57
58	${essay}`
59
60	const getEssays = async (): Promise<Essay[]> => {	18✔
61	const postIds = await ReviewWinners	×
62	.find({}, { projection: { postId: 1 }, sort: { reviewRanking: 1 } })
63	.fetch();
64	const reviewArts = await ReviewWinnerArts	×
65	.find({})
66	.fetch();
67	const postIdsWithoutLotsOfArt = postIds	×
68	.filter(p => reviewArts.filter(a => a.postId === p.postId).length < 9)	×
69	const postIdsWithoutEnoughArt = postIds	×
70	.filter(p => reviewArts.filter(a => a.postId === p.postId).length < 10)	×
71
72	const postsToFind = postIdsWithoutLotsOfArt.length > 0 ? postIdsWithoutLotsOfArt : postIdsWithoutEnoughArt	×
73
NEW 74	const essays = await fetchFragment({	×
75	collectionName: "Posts",
76	fragmentName: "PostsPage",
NEW 77	selector: {_id: {$in: postsToFind.map(p => p.postId)}},	×
78	currentUser: null,
79	skipFiltering: true,
80	});
81
NEW 82	const toGenerate = (p: PostsPage) => Math.ceil((12 - reviewArts.filter(art => art.postId === p._id).length)/4)	×
83
NEW 84	return essays.map(e => {	×
85	return {post: e, title: e.title, content: e.contents?.html ?? "", toGenerate: toGenerate(e) }	×
86	})
87	}
88
89	type Essay = {post: PostsPage, title: string, content: string, toGenerate: number}
90	type MyMidjourneyResponse = {messageId: "string", uri?: string, progress: number, error?: string}
91
92	const getElements = async (openAiClient: OpenAI, essay: {title: string, content: string}, tryCount = 0): Promise<string[]> => {	18!
93	const content = essay.content.length > 25_000 ? essay.content.slice(0, 12_500) + "\n[EXCERPTED FOR LENGTH]\n" + essay.content.slice(-12_500) : essay.content	×
94	const completion = await openAiClient.chat.completions.create({	×
95	messages: [{role: "user", content: llm_prompt(essay.title, content)}],
96	model: "gpt-4",
97	}).catch((error) => {
98	if (error instanceof OpenAI.APIError && error.status === 400 && error.code === 'context_length_exceeded') {	×
99	return openAiClient.chat.completions.create({	×
100	messages: [{role: "user", content: llm_prompt(essay.title, content.slice(0, 8_000) + "\n[EXCERPTED FOR LENGTH]\n" + essay.content.slice(-8_000))}],
101	model: "gpt-4",
102	})
103	} else {
104	// eslint-disable-next-line no-console
105	console.error(error)	×
106	return undefined	×
107	}
108	})
109
110
111	try {	×
112	return JSON.parse((completion?.choices[0].message.content \|\| '').split('METAPHORS: ')[1])	×
113	} catch (error) {
114	// eslint-disable-next-line no-console
115	console.error('Error parsing response:', error);	×
116	if (tryCount < 2) return getElements(openAiClient, essay, tryCount + 1)	×
117	return []	×
118	}
119	}
120
121	const prompter = (el: string) => {	18✔
122	const lowerCased = el[0].toLowerCase() + el.slice(1)	×
123	return `${shuffle(promptUrls)[0]} topographic watercolor artwork of ${lowerCased}, in the style of ethereal watercolor washes, ultrafine detail, juxtaposition of hard and soft lines, delicate ink lines, inspired by scientific illustrations, in the style of meditative pastel moebius, muted colors --ar 8:5 --v 6.0 `	×
124	}
125
126	const pressMidjourneyButton = async (messageId: string, button: string) => {	18✔
127	return fetch(`https://api.mymidjourney.ai/api/v1/midjourney/button`, {	×
128	method: 'POST',
129	headers: {
130	'Authorization': `Bearer ${myMidjourneyKey}`,
131	'Content-Type': 'application/json'
132	},
133	body: JSON.stringify({messageId, button})
134	}).then(r => r.json())	×
135	.then(j => j.messageId)	×
136	.then(checkOnJob)
137	}
138
139	const saveImage = async (el: string, essay: Essay, url: string) => {	18✔
140	const newUrl = await moveImageToCloudinary(url, `splashArtImagePrompt${el}`)	×
141	if (!newUrl) {	×
142	// eslint-disable-next-line no-console
143	console.error("Failed to upload image to cloudinary", el, essay)	×
144	return	×
145	}
146	await createMutator({	×
147	collection: ReviewWinnerArts,
148	context: createAdminContext(),
149	document: {
150	postId: essay.post._id,
151	splashArtImagePrompt: el,
152	splashArtImageUrl: newUrl
153	}
154	})
155	return newUrl	×
156	}
157
158	const upscaledImages = async (el: string, essay: Essay, messageId: string): Promise<(string \| undefined)[]> =>	18✔
159	Promise.all(["U1","U2","U3","U4"]	×
160	.map(async button => {
161	return pressMidjourneyButton(messageId, button)	×
162	.then(m => m && upscaleImage(m.messageId))	×
163	.then(uri => uri && saveImage(el, essay, uri))	×
164	}))
165
166	const upscaleImage = async (messageId: string): Promise<string \| undefined> => {	18✔
167	const res = await pressMidjourneyButton(messageId, "Upscale (Subtle)")	×
168	return res?.uri	×
169	}
170
171	async function checkOnJob(jobId: string): Promise<MyMidjourneyResponse \| undefined> {
172	try {	×
173	const response = await fetch(`https://api.mymidjourney.ai/api/v1/midjourney/message/${jobId}`, {	×
174	method: 'GET',
175	headers: {
176	'Authorization': `Bearer ${myMidjourneyKey}`,
177	'Content-Type': 'application/json'
178	}
179	})
180
181	const responseData = await response.json()	×
182	if (responseData.progress === 100 \|\| responseData.error) {	×
183	if (responseData.error) {	×
184	// eslint-disable-next-line no-console
185	console.error('Image generation failed:', responseData);	×
186	return undefined	×
187	}
188	return responseData	×
189	} else {
190	await sleep(5_000)	×
191	return checkOnJob(jobId)	×
192	}
193	} catch (error) {
194	// eslint-disable-next-line no-console
195	console.error('Error getting updates', error);	×
196	return checkOnJob(jobId)	×
197	}
198	}
199
200	async function getEssayPromptJointImageMessage(promptElement: string): Promise<MyMidjourneyResponse\|undefined> {
201	let promptResponseData: any;
202	let jobId: string;
203	try {	×
204	const response = await fetch('https://api.mymidjourney.ai/api/v1/midjourney/imagine', {	×
205	method: 'POST',
206	headers: {
207	'Authorization': `Bearer ${myMidjourneyKey}`,
208	'Content-Type': 'application/json'
209	},
210	body: JSON.stringify({prompt: prompter(promptElement)})
211	});
212
213	promptResponseData = await response.json();	×
214	jobId = promptResponseData.messageId	×
215	return checkOnJob(jobId)	×
216	} catch (error) {
217	// eslint-disable-next-line no-console
218	console.error('Error generating image:', error);	×
219	}
220
221	}
222
223	async function generateCoverImages({limit = 2}: {	×
224	limit?: number
225	} = {}): Promise<string[]> {	×
226	const openAiClient = await getOpenAI()	×
227	if (!openAiClient) {	×
228	throw new Error('Could not initialize OpenAI client!');	×
229	}
230	const essays = (await getEssays()).slice(0, limit)	×
231
232	return await essays.reduce(async (prev: Promise<string[]>, essay: Essay): Promise<string[]> => {	×
233
234	const prevUrls = await prev	×
235	const images = await getElements(openAiClient, essay)	×
236	.then((els: string[]) =>
237	els.slice(0,Math.min(limit, essay.toGenerate))	×
238	.reduce(async (prev, el) => {
239	const ims = await prev	×
240	const im = await getEssayPromptJointImageMessage(el)	×
241	.then(x => x === undefined ? Promise.resolve([]) : upscaledImages(el, essay, x.messageId))	×
242	.then(urls => filterNonnull(urls))	×
243	return [...ims, ...im]	×
244	}, Promise.resolve([]) as Promise<string[]>))
245
246	return [...prevUrls, ...images]	×
247	}, Promise.resolve([]) as Promise<string[]>)
248	}
249
250	async function coverImages () {
251	if (!myMidjourneyKey) {	×
252	throw new Error('No MyMidjourney API key found!');	×
253	}
254
255	const newImages = await generateCoverImages({limit: 9999})	×
256	if (newImages.length === 0) return	×
257	await sleep(10_000)	×
258	// There are often failures in the process of the generation. We run until we no longer have candidate essays
259	await coverImages()	×
260	}
261
262	Globals.coverImages = coverImages	18✔

ForumMagnum / ForumMagnum / 9553205131

Source File Press 'n' to go to next uncovered line, 'b' for previous

Source File
Press 'n' to go to next uncovered line, 'b' for previous